The robots.txt testing tool in Google Webmaster Tools has just received an update to highlight errors causing Google not to crawl pages on your website, let you edit your file, test if URLs are blocked, and let you view older versions of your file.
If Google isn’t crawling a page or part of your site, the robots.txt tester, located under the Crawl section of Google Webmaster Tools, will now let you test whether there’s an issue in your file that’s blocking Google. (This section of GWT used to be known as Blocked URLs.)
“To guide your way through complicated directives, it will highlight the specific one that led to the final decision,” wrote Asaph Arnon, Webmaster Tools team on the Google Webmaster Central blog. “You can make changes in the file and test those too, you’ll just need to upload the new version of the file to your server afterwards to make the changes take effect.
“Additionally, you’ll be able to review older versions of your robots.txt file, and see when access issues block us from crawling,” he continued. “For example, if Googlebot sees a 500 server error for the robots.txt file, we’ll generally pause further crawling of the website.”
In announcing the news on Google+, Google’s John Mueller wrote that even if you think your robots.txt is fine, you should check for errors or warnings.
“Some of these issues can be subtle and easy to miss,” he wrote. “While you’re at it, also double-check how the important pages of your site render with Googlebot, and if you’re accidentally blocking any JS or CSS files from crawling.”