Search Engine Watch
SEO News

Go Back   Search Engine Watch Forums > Search Engine Marketing Strategies > Search Engine Optimization > Dynamic Website and Technical Issues
FAQ Members List Calendar Forum Search Today's Posts Mark Forums Read

Reply
 
Thread Tools
Old 01-16-2008   #1
CaliforniaGirl
Member
 
Join Date: Sep 2005
Location: Sydney, Australia
Posts: 42
CaliforniaGirl is on a distinguished road
Block IP ranges of Google, MSN and Yahoo

G'day everyone,

I have a client who uses 30+ servers for site testing and an even bigger amount of domain names. Due to this engines are indexing mirrored copies of the preferred domain (testing sites each have a different URL). Because of the amount of sites and other issues, options such as robots.txt files and password/security measurements are too resource heavy and/or restrictive. My advice is to block the IP ranges of the majors (Google, Yahoo and MSN at least).

I have identified Google's IP range:
64.233.160.0 - 64.233.191.255

I'd like advice on the following:

1. Is the Google IP range correct?
2. Can you provide the ranges for MSN/Live and Yahoo?
3. Is there a better method that I have missed?

Thanks in advance
CaliforniaGirl
CaliforniaGirl is offline   Reply With Quote
Old 01-16-2008   #2
AussieWebmaster
Forums Editor, SearchEngineWatch
 
AussieWebmaster's Avatar
 
Join Date: Jun 2004
Location: NYC
Posts: 8,153
AussieWebmaster has a brilliant futureAussieWebmaster has a brilliant futureAussieWebmaster has a brilliant futureAussieWebmaster has a brilliant futureAussieWebmaster has a brilliant futureAussieWebmaster has a brilliant futureAussieWebmaster has a brilliant futureAussieWebmaster has a brilliant futureAussieWebmaster has a brilliant futureAussieWebmaster has a brilliant futureAussieWebmaster has a brilliant future
Re: Block IP ranges of Google, MSN and Yahoo

okay cali girl... much prefer Byron... but Brisbane
is my home
AussieWebmaster is offline   Reply With Quote
Old 01-16-2008   #3
jimbeetle
 
jimbeetle's Avatar
 
Join Date: Mar 2006
Location: New York City
Posts: 997
jimbeetle has much to be proud ofjimbeetle has much to be proud ofjimbeetle has much to be proud ofjimbeetle has much to be proud ofjimbeetle has much to be proud ofjimbeetle has much to be proud ofjimbeetle has much to be proud ofjimbeetle has much to be proud ofjimbeetle has much to be proud of
Re: Block IP ranges of Google, MSN and Yahoo

Dan has a pretty complete IP lists.
jimbeetle is offline   Reply With Quote
Old 01-16-2008   #4
CaliforniaGirl
Member
 
Join Date: Sep 2005
Location: Sydney, Australia
Posts: 42
CaliforniaGirl is on a distinguished road
Re: Block IP ranges of Google, MSN and Yahoo

Thanks guys. The IP lists are great (and huge). Can I presume that this method is the best option in your opinion?

CaliforniaGirl

~Byron is great - my heart lies with Sydney's Northern Beaches
CaliforniaGirl is offline   Reply With Quote
Old 01-18-2008   #5
btard
affiliate/seo ninja
 
Join Date: Jan 2008
Location: vancouver
Posts: 45
btard is on a distinguished road
Re: Block IP ranges of Google, MSN and Yahoo

Quote:
Originally Posted by jimbeetle View Post
Dan has a pretty complete IP lists.
there's not really a complete list now is there... ps. that was not a question. the reason i say this is because you may think you're blocking all IPs but you're not.

also bear in mind that google will see a referrer and will eventually find a dupe site. the chances are of course minimized then again, don't rely on it 100%
btard is offline   Reply With Quote
Old 01-20-2008   #6
Jazajay
 
Jazajay's Avatar
 
Join Date: Jul 2007
Location: Leicester, England
Posts: 713
Jazajay has disabled reputation
Re: Block IP ranges of Google, MSN and Yahoo

Hay up
How about testing the user-agent for googlebot?
Um I suppose it's still a problem when they come through a proxy?
Does G still use proxies? Um...
There's my 2 cents anyway sorry it's not great and slightly contradictory .
Jazajay is offline   Reply With Quote
Old 01-20-2008   #7
btard
affiliate/seo ninja
 
Join Date: Jan 2008
Location: vancouver
Posts: 45
btard is on a distinguished road
Re: Block IP ranges of Google, MSN and Yahoo

Quote:
Originally Posted by Jazajay View Post
Hay up
How about testing the user-agent for googlebot?
Um I suppose it's still a problem when they come through a proxy?
Does G still use proxies? Um...
There's my 2 cents anyway sorry it's not great and slightly contradictory .
again that will help but not cover all bases. after spending two years working in the IM dept of a huge online gaming company and having millions of dollars to spend we decided that gaming google was not an effective way to make a good internet presence.

if you're in this for "test" purposes then why do you care if google catches you duping content?

now stop contradicting me boy
btard is offline   Reply With Quote
Old 01-20-2008   #8
Jazajay
 
Jazajay's Avatar
 
Join Date: Jul 2007
Location: Leicester, England
Posts: 713
Jazajay has disabled reputation
Re: Block IP ranges of Google, MSN and Yahoo

Yeah I was clutching at straws to be honest.
I'll think on it but to be honest I cant see a perfect solution sorry, especially recently with G saying no matter what you do we will detect cloaking. However some sites with #1 SERPS are still showing cloaking. Same they don't use noarchive I would hate to pay

What if you test the user-agent. Then if the user-agent doesn't = googlebot test the IP as well. Hum....

To be honest 30 servers is quite a lot why not cut it down and use A-B testing over X amount of IP's each? Surely you would get the same results if you ran the test for say 6-8 months with 10,000 unquie IP's, then look at the conversions or what ever. That would give you a % increase overall or a dead heat.

Plus you would save 29 servers. I cant see why you need so many servers just for testing even for a big corporation.

I mean if you had 1/country surly the company you work for would have a domain name /country thus getting rid of this issue?

Would 30 servers give you fair test results anyway? Surly there would be too many variables to say it was a fair test. If so why 30 servers? You could accomplish the same with 1 or 2.

If it is usability you are looking at it would be cheaper to high a professional for a few days and say 1000 test subjects and let them tell you the results of how usable your site(s) are.

I have a total of 6 servers and I run mine via an array. I list the Ip's if one goes off line the content/image is delivered via one of the other 5. The main pages are delivered via the same main server which is more powerful than the other 5. Hasn't gone off line yet. While the images are brought in via 3 to maximise my HTTP paraliziation download time. Tracking done by the 4th. security, firewall, secure session data in the DB done by the 5th and the 6th is used as a Backup/Golden server. All servers have the same images, pages and DB content so they can take over from their main jobs if 1 of the other ones delivers a 404.

When I do testing I remove one of the smaller servers/tends to be the 6th and reroute all the testing data to that one dedicated server via a Constant that changes my server logic.

Just an idea. I'm missing key data though -
what you are testing,
expected results,
constants
variables
etc...
So it may not be practical for your situation.
Jaz
Jazajay is offline   Reply With Quote
Reply


Currently Active Users Viewing This Thread: 1 (0 members and 1 guests)
 
Thread Tools

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

vB code is On
Smilies are On
[IMG] code is On
HTML code is Off