Some Robots.txt Standards of Microsoft, Yahoo! and Google

by buxing on 2008-06-19 22:19:47

The three major search engines are having a great time competing, but they also collaborate occasionally. Last year, Google, Yahoo, and Microsoft collaborated by adhering to a unified Sitemaps standard. A few days ago, the three giants announced again that they will jointly comply with the robots.txt file standard. Google, Yahoo, and Microsoft each posted an article on their respective official blogs, announcing the standards for robots.txt files and Meta tags supported by all three, as well as some individual-specific standards. Below is a summary.