A common standard for the Robots.txt protocol
Microsoft, Yahoo and Google have developed a detailed documentation about how they implement the Robots Exclusion Protocol (REP).
The search engines’ robots go wherever they want as long as they find links to follow. This means that they may index everything they find on your site or server unless you tell them explicitly not to.
There […]
