NewComputer
09-27-2004, 12:05 AM
It looks as though a mod rewrite was being used for error pages. My question is, if the spiders come looking for the robots.txt file and there is not one, what happens. If you go to www.mydomain.com/robots.txt you are sent to the home page, I wonder if that is not a good thing.
seobook
09-27-2004, 12:29 AM
usually they will just spider your site.
it is recommended to use a blank one versus none at all.
Hank Cowdog
02-04-2005, 04:22 PM
Unless the home page just happens to have text in a valid format for the Robots Exclusion Standard (unlikely), the site will be spidered normally.
That being said. creating a blank robots.txt solves a few minor annoyances:
- unnecessary 404 errors in the access and error logs
- extra processing for the spider to read the large (compared to the typical robots.txt file) amount of text returned.