Has anybody come up with a good robots.txt file? I wouldn’t know what should be disallowed.
HI Dannyboy, were you able to locate one?
No, not yet. I am still hopeful someone with knowledge in this area will share their version of robots.txt, or at least discuss it.
I don’t know much about robot.txt files. I however did a bit of research and came across a few artilcles that I think might be helpful.
This one has some good examples about what to disallow, it also has some links to robot.txt file generators:
Here are some other links I found useful: