Robots.txt

Google just introduced a new robots.txt tool (in Sitemaps).

The robots.txt file commonly has mistakes.

Even with good information on the web, experts have trouble knowing what a certain robots.txt will do. If you get the file wrong, it can have a big impact on your site, which makes people nervous (rightfully so).  Every search engine has slightly different extra options that they support in the robots.txt file.

The nice thing about the robots.txt checker in Sitemaps is that it lets you test a robots.txt file and see how the real Googlebot would handle it.

Popularity: 7% [?]

If you enjoyed this post, please consider to leave a comment or subscribe to the feed and get future articles delivered to your feed reader.