Search Engine Watch
SEO News

Go Back   Search Engine Watch Forums > Search Engines & Directories > Google > Other Google Issues
FAQ Members List Calendar Forum Search Today's Posts Mark Forums Read

Reply
 
Thread Tools
Old 08-30-2007   #1
pstewart
Newbie
 
Join Date: Aug 2007
Location: new york city
Posts: 1
pstewart is on a distinguished road
Is robot.txt file a must requirement?

I keep on receiving error from google that states that my page is invalid in robot.txt file. The thing is - I don't have robot.txt file. So, my question is - do I need to have one? My sitemap.xml file lists all files that need to be indexed.

Thanks,
Paul Stewart

Last edited by JohnW : 08-30-2007 at 08:12 PM. Reason: removed link
pstewart is offline   Reply With Quote
Old 08-30-2007   #2
jimbeetle
 
jimbeetle's Avatar
 
Join Date: Mar 2006
Location: New York City
Posts: 997
jimbeetle has much to be proud ofjimbeetle has much to be proud ofjimbeetle has much to be proud ofjimbeetle has much to be proud ofjimbeetle has much to be proud ofjimbeetle has much to be proud ofjimbeetle has much to be proud ofjimbeetle has much to be proud ofjimbeetle has much to be proud of
Re: Is robot.txt file a must requirement?

No, it isn't necessary to have a robots.txt file, though you might consider putting up a blank robots.txt file so you can rid yourself of those 404 errors. Simply open a new file in Notepad or other text editor, save it as robots.txt and upload it to the www root of your server.

The original robots.txt standard has been extended by both Google and Yahoo, so if you do decide to use one, be sure to familiarize yourself with the wildcard (*) and end anchor ($) symbols. I usually recommend against using these as only G and Y are known to recognize them at this time.
jimbeetle is offline   Reply With Quote
Reply


Currently Active Users Viewing This Thread: 1 (0 members and 1 guests)
 
Thread Tools

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

vB code is On
Smilies are On
[IMG] code is On
HTML code is Off

Similar Threads
Thread Thread Starter Forum Replies Last Post
google site map xml file validation issues with URLs outofbounds Dynamic Website and Technical Issues 3 08-11-2007 03:55 PM
File To File Redirect gregs Search Engine Optimization 0 11-15-2006 04:20 PM
Optimizing Flash files for the search engines rockcoastmedia Search Engine Optimization 30 04-11-2006 07:40 PM
Changing default file names gdawg Search Engine Optimization 1 10-07-2005 01:58 PM
Xml File Issues? critter Search Engine Optimization 1 12-17-2004 12:04 PM