Robots.txt


#1

Has anybody come up with a good robots.txt file? I wouldn’t know what should be disallowed.


#2

HI Dannyboy, were you able to locate one?

Heather


#3

No, not yet. I am still hopeful someone with knowledge in this area will share their version of robots.txt, or at least discuss it.


#4

`Hey Dannyboy,

I don’t know much about robot.txt files. I however did a bit of research and came across a few artilcles that I think might be helpful.

This one has some good examples about what to disallow, it also has some links to robot.txt file generators:

http://www.outfront.net/tutorials_02/adv_tech/robots.htm

Here are some other links I found useful:

http://support.google.com/webmasters/bin/answer.py?hl=en&answer=156449&from=35237&rd=1

http://www.buildbrandbelieve.com/robots-txt-file-explained

http://www.koozai.com/blog/search-marketing/a-brief-guide-to-robots-txt-and-five-mistakes-to-avoid/

Regards,

Sharon.