Thanks Yahoo blog, I have discovered Web Robots Pages, about Robot.txt file. Web Robots are programs that traverse the Web automatically. Some people call them Web Wanderers, Crawlers, or Spiders. Thanks Robotstxt.org, you can have further information about these web robots :

  • Web Robots FAQ
  • Robots Exclusion
  • A list of Robots
  • Robots mailing lists (with technical forum)
  • Main big articles about Robots
  • Related websites (I notice especially Robogen)

In my opinion, this website is a really good help to understand why robots exists. FAQ is really good, short and the most important / relevant is here. If it is not enough, just read one of articles about web robots.