View Full Version : Removing cahced robots.txt file from Google Index
Jeff Martin
04-19-2007, 04:21 PM
I hadnt seen a cahced robots.txt file in a long time and I saw another today for a site of mine. Does anyone know if there are any concerns with removing the robots.txt URL from the Webmaster Tools console?
jimbeetle
04-20-2007, 10:43 AM
I have no idea what the implications of that would be, don't think I'd what to find out.
You might want to try the usual stuff first: validate the robots.txt, and use a server header utility to check that the file is being served as text and not some other type.
Jeff Martin
04-20-2007, 11:00 AM
This is not a unique instance. Google has quite a few of these cached for other sites. It is not a server issue.
jimbeetle
04-20-2007, 05:08 PM
I'd never noticed an indexed robots.txt file before. Couldn't find any earlier, but now that I run [allinurl:"com/robots.txt"] on Google there's many more than I would expect that are indexed and cached. Can't find any similarities among the files that would cause this. Must be a Google thing as I don't see the same behaviour by Y!.