Has anybody come up with a good robots.txt file? I wouldn’t know what should be disallowed.


HI Dannyboy, were you able to locate one?



No, not yet. I am still hopeful someone with knowledge in this area will share their version of robots.txt, or at least discuss it.


`Hey Dannyboy,

I don’t know much about robot.txt files. I however did a bit of research and came across a few artilcles that I think might be helpful.

This one has some good examples about what to disallow, it also has some links to robot.txt file generators:

Here are some other links I found useful: