Join Date: Aug 2007
Location: new york city
Is robot.txt file a must requirement?
I keep on receiving error from google that states that my page is invalid in robot.txt file. The thing is - I don't have robot.txt file. So, my question is - do I need to have one? My sitemap.xml file lists all files that need to be indexed.
Last edited by JohnW : 08-30-2007 at 09:12 PM. Reason: removed link
Oversees: Dynamic Website and Technical Issues
Join Date: Mar 2006
Location: New York City
Re: Is robot.txt file a must requirement?
No, it isn't necessary to have a robots.txt file, though you might consider putting up a blank robots.txt file so you can rid yourself of those 404 errors. Simply open a new file in Notepad or other text editor, save it as robots.txt and upload it to the www root of your server.
The original robots.txt standard has been extended by both Google and Yahoo, so if you do decide to use one, be sure to familiarize yourself with the wildcard (*) and end anchor ($) symbols. I usually recommend against using these as only G and Y are known to recognize them at this time.
|Currently Active Users Viewing This Thread: 1 (0 members and 1 guests)|
|Thread||Thread Starter||Forum||Replies||Last Post|
|google site map xml file validation issues with URLs||outofbounds||Dynamic Website and Technical Issues||3||08-11-2007 04:55 PM|
|File To File Redirect||gregs||Search Engine Optimization||0||11-15-2006 05:20 PM|
|Optimizing Flash files for the search engines||rockcoastmedia||Search Engine Optimization||30||04-11-2006 08:40 PM|
|Changing default file names||gdawg||Search Engine Optimization||1||10-07-2005 02:58 PM|
|Xml File Issues?||critter||Search Engine Optimization||1||12-17-2004 01:04 PM|