404 errors and robots.txt file

[newreality]

Looking at my stats for a couply different sites I'm noticing many 404's for the robots.txt file.

Isn't this the one the search engines search for first?

Is it really important that the file be present. I'm not using ROBOTS in any of the page Meta's.

[mcanerin]

It won't hurt you if it's not there, but I recommend putting one in to clean up the errors from your logs.

You can even put in a totally blank text file called robots.txt and it will do the job.

Ian

[David Wallace]

Robots meta tags do not make any difference and that is not what SEs are looking for. They simply look for the robots.txt file first to see if there any restrictions being placed on them as far as what they can and cannot crawl. You don't need one but as Ian said, if you add one, then your 404's will disappear.

[newreality]

Thank you Ian and David. Nice to hear, I've got enough to do right now.