Robots.txt file – Do I Need It?
A robots.txt file basically tells search engines which part of your site not to crawl. The format is to put it in your top level directory in the format. It is useful in restricting parts of your site you do not want indexed by certain search engines. Expamples include:
User-agent: * Disallow:
Examples of Other formats include the following
Disallow All crawlers
User-agent: * Disallow: /
Restrict folder
User-agent: *
Disallow: /private/
Restrict file to all robots
User-agent: * Disallow: /directory/file.html
Some may wonder whether it is worth having a robots.txt file at all if you want to give search engine crawlers unrestricted access. Matt Cutts has answered in his weekly digest that it is useful to have the files even if it is to say disallow none.
