Search Engine Watch
SEO News

 

  #1  
Old 11-03-2005
onedodd onedodd is offline
Onedodd
 
Join Date: Jan 2005
Posts: 99
onedodd is on a distinguished road
Robots.txt Question

Will Google read and follow all of this robots.txt file?

Will Google block Page 1 thru 6 ? This * means all bots right?
I am wondering if I add User-agent: googlebot does that over-ride this * and mean that Google will only adhere to User-agent: googlebot instructions or will it still follow * instructions too?

The idea is I want Google blocked from all pages listed and MSN and Yahoo blocked only on page-1.html, page-2.html, and page-3.html.


User-agent: *
Disallow: /page-1.html
Disallow: /page-2.html
Disallow: /page-3.html

User-agent: googlebot
Disallow: /page-4.html
Disallow: /page-5.html
Disallow: /page-6.html

OK, I have confused myself now and if anyone can decipher my question I'd appraciate any help.

Thanks, Joe
Reply With Quote
  #2  
Old 11-03-2005
bhartzer bhartzer is offline
Search Engine Optimization, Search Engine Marketing Expert
 
Join Date: Jun 2004
Location: Dallas, Texas
Posts: 539
bhartzer has a spectacular aura aboutbhartzer has a spectacular aura aboutbhartzer has a spectacular aura about
Quote:
This * means all bots right?
Yes.

If you want to target certain engines for certain pages then don't use the * anywhere. Be very specific with the instructions for each bot.
__________________
Bill Hartzer is an internet marketing consultant in Dallas and has been practicing organic SEO since 1996.
Reply With Quote
Reply


Currently Active Users Viewing This Thread: 1 (0 members and 1 guests)
 
Thread Tools

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

vB code is On
Smilies are On
[IMG] code is On
HTML code is Off
Forum Jump


All times are GMT -4. The time now is 10:32 AM.