|
#1
|
|||
|
|||
|
Robots.txt Question
Will Google read and follow all of this robots.txt file?
Will Google block Page 1 thru 6 ? This * means all bots right? I am wondering if I add User-agent: googlebot does that over-ride this * and mean that Google will only adhere to User-agent: googlebot instructions or will it still follow * instructions too? The idea is I want Google blocked from all pages listed and MSN and Yahoo blocked only on page-1.html, page-2.html, and page-3.html. User-agent: * Disallow: /page-1.html Disallow: /page-2.html Disallow: /page-3.html User-agent: googlebot Disallow: /page-4.html Disallow: /page-5.html Disallow: /page-6.html OK, I have confused myself now and if anyone can decipher my question I'd appraciate any help. Thanks, Joe |
|
#2
|
|||
|
|||
|
Quote:
If you want to target certain engines for certain pages then don't use the * anywhere. Be very specific with the instructions for each bot.
__________________
Bill Hartzer is an internet marketing consultant in Dallas and has been practicing organic SEO since 1996. |
![]() |
| Currently Active Users Viewing This Thread: 1 (0 members and 1 guests) | |
| Thread Tools | |
|
|