Search Engine Watch
SEO News

Go Back   Search Engine Watch Forums > General Search Issues > Searching Tips & Techniques
FAQ Members List Calendar Forum Search Today's Posts Mark Forums Read

Reply
 
Thread Tools
Old 08-31-2004   #1
rustybrick
 
rustybrick's Avatar
 
Join Date: Jun 2004
Location: New York, USA
Posts: 2,810
rustybrick has much to be proud ofrustybrick has much to be proud ofrustybrick has much to be proud ofrustybrick has much to be proud ofrustybrick has much to be proud ofrustybrick has much to be proud ofrustybrick has much to be proud ofrustybrick has much to be proud of
How Do I Spot Cloaked Sites?

Forget the debate about cloaking, I am a bit tired of that anyway. How does one detect some of the cloaking going on around the Web. Follow these instructions:

(1) Download the Firefox Browser
(2) Install it
(3) Download the User Agent Switcher for Firefox/Mozilla while using firefox
(4) Restart the browser
(5) Under Tools --> User Agent Switcher --> Options --> Options (that will open a dialog box)
(6) Click Add Under User Agents section
(7) In the description add "Googlebot" and in the user agent add "Googlebot/2.1 (+http://www.googlebot.com/bot.html)"
(8) Repeat this process for all the spiders you want to test. Updated comprehensive list of user agents.
(9) Under Tools --> User Agent Switcher --> select the user agent
(10) Then navigate to the pages that you want to test for cloaking.

Hope this helps some people be Googlebot.
rustybrick is offline   Reply With Quote
Old 08-31-2004   #2
Nick W
Member
 
Join Date: Jun 2004
Posts: 593
Nick W is a jewel in the roughNick W is a jewel in the roughNick W is a jewel in the roughNick W is a jewel in the rough
Not bad Rusty.

Unfortunately that's only going to catch the incompetent cloakers. Most cloakers wouldnt give a stuff about UA's, they care about IP's

Shame firefox is so 'different' to Moz, some of the stuff is really round the wrong way, i just cant get used to it...

Nick
Nick W is offline   Reply With Quote
Old 08-31-2004   #3
mcanerin
 
mcanerin's Avatar
 
Join Date: Jun 2004
Location: Calgary, Alberta, Canada
Posts: 1,564
mcanerin has a reputation beyond reputemcanerin has a reputation beyond reputemcanerin has a reputation beyond reputemcanerin has a reputation beyond reputemcanerin has a reputation beyond reputemcanerin has a reputation beyond reputemcanerin has a reputation beyond reputemcanerin has a reputation beyond reputemcanerin has a reputation beyond reputemcanerin has a reputation beyond reputemcanerin has a reputation beyond repute
Unfortunately, although it's easy to spoof an IP one way, it's very hard to spoof an IP and actually get a response back (the response would go to the spoofed IP, not yours).

There are ways around this but it's pretty complex, and usually not worth it.

I wonder how often (if ever) Google employees go home with a bunch of websites and visit them from there That would be really funny. Heck, you could even have a separate line/ISP coming into Googleplex just for surfing the web on non-googlebot IP's. Would be easy if they wanted to. Could even write a script to automatically compare a site that had been flagged.

That would be lots of fun, too.....

Yes, I'm evil.....

Ian
__________________
International SEO

Last edited by mcanerin : 08-31-2004 at 01:34 PM. Reason: Typos are my lifes work....
mcanerin is offline   Reply With Quote
Old 08-31-2004   #4
hiero
If winning isn't everything, why do they keep score? --Vince Lombardi
 
Join Date: Aug 2004
Location: Los Angeles, California
Posts: 119
hiero is on a distinguished road
Smile

Pretty cool Barry......anyone have a url to test?
hiero is offline   Reply With Quote
Old 08-31-2004   #5
rustybrick
 
rustybrick's Avatar
 
Join Date: Jun 2004
Location: New York, USA
Posts: 2,810
rustybrick has much to be proud ofrustybrick has much to be proud ofrustybrick has much to be proud ofrustybrick has much to be proud ofrustybrick has much to be proud ofrustybrick has much to be proud ofrustybrick has much to be proud ofrustybrick has much to be proud of
Quote:
Originally Posted by Nick W
Not bad Rusty.

Unfortunately that's only going to catch the incompetent cloakers. Most cloakers wouldnt give a stuff about UA's, they care about IP's

Shame firefox is so 'different' to Moz, some of the stuff is really round the wrong way, i just cant get used to it...

Nick
Nick, 100% right. But its funny to see those people who cloak without the IP delivery methods. Of course mcanerin idea would work, but you would need some powerful software thats updated frequently to find these sites that way. And then, if your doing that, you can then easily cloak yourself.
rustybrick is offline   Reply With Quote
Old 08-31-2004   #6
hiero
If winning isn't everything, why do they keep score? --Vince Lombardi
 
Join Date: Aug 2004
Location: Los Angeles, California
Posts: 119
hiero is on a distinguished road
Arrow

Here is a link to some cloaking examples:
http://fantomaster.com/fafaqcloak4.html

Using their first example:
Now if I'm using Firefox with the user-agent set to googlebot I should be seeing the bot page, but I'm not.

So I can only assume that the cgi script doesn't include googlebot 2.1 as an agent, right?
hiero is offline   Reply With Quote
Old 08-31-2004   #7
rustybrick
 
rustybrick's Avatar
 
Join Date: Jun 2004
Location: New York, USA
Posts: 2,810
rustybrick has much to be proud ofrustybrick has much to be proud ofrustybrick has much to be proud ofrustybrick has much to be proud ofrustybrick has much to be proud ofrustybrick has much to be proud ofrustybrick has much to be proud ofrustybrick has much to be proud of
Um, well no. Fantomaster's products are a bit more sophisticated then simply changing the user agent. They use IP Delivery. They know Googlebot's IP addresses and based on that info, they redirect the bot to a different page. What Nick said above in post # 2.
rustybrick is offline   Reply With Quote
Old 08-31-2004   #8
NFFC
"One wants to have, you know, a little class." DianeV
 
Join Date: Jun 2004
Posts: 468
NFFC is a splendid one to beholdNFFC is a splendid one to beholdNFFC is a splendid one to beholdNFFC is a splendid one to beholdNFFC is a splendid one to beholdNFFC is a splendid one to behold
>they redirect the bot to a different page

I think the word redirect isn't quite right.

It seems to me that they provide a customised experience according to user expectations.
NFFC is offline   Reply With Quote
Old 08-31-2004   #9
hiero
If winning isn't everything, why do they keep score? --Vince Lombardi
 
Join Date: Aug 2004
Location: Los Angeles, California
Posts: 119
hiero is on a distinguished road
Got it, thanks.

Then what real value do you see the user agent switcher for Firefox having? Is it a worthwhile add on?
hiero is offline   Reply With Quote
Old 08-31-2004   #10
Nick W
Member
 
Join Date: Jun 2004
Posts: 593
Nick W is a jewel in the roughNick W is a jewel in the roughNick W is a jewel in the roughNick W is a jewel in the rough
It would help with those dumb arsed sites that insist you get IE b4 u can view their crappy AOL like pages.......

...as if you'd want to eh?

Nick
Nick W is offline   Reply With Quote
Old 08-31-2004   #11
rustybrick
 
rustybrick's Avatar
 
Join Date: Jun 2004
Location: New York, USA
Posts: 2,810
rustybrick has much to be proud ofrustybrick has much to be proud ofrustybrick has much to be proud ofrustybrick has much to be proud ofrustybrick has much to be proud ofrustybrick has much to be proud ofrustybrick has much to be proud ofrustybrick has much to be proud of
There are many sites that deploy user agent based cloaking only. Why? I guess because its easier.
rustybrick is offline   Reply With Quote
Old 08-31-2004   #12
hiero
If winning isn't everything, why do they keep score? --Vince Lombardi
 
Join Date: Aug 2004
Location: Los Angeles, California
Posts: 119
hiero is on a distinguished road
Arrow

So if I write a javascript routine that looks at the user agent and returns a page for that user agent that would be cloaking also. hmmmm.....weird.
hiero is offline   Reply With Quote
Old 08-31-2004   #13
seomike
Md_Rewrite Guru
 
Join Date: Jun 2004
Location: Dallas, Texas but forever a Floridian!
Posts: 627
seomike is a splendid one to beholdseomike is a splendid one to beholdseomike is a splendid one to beholdseomike is a splendid one to beholdseomike is a splendid one to beholdseomike is a splendid one to beholdseomike is a splendid one to behold
Go to webmasterworld.com if you have a user agent with a search engine bot name you'll see meta tags otherwise you won't
seomike is offline   Reply With Quote
Old 08-31-2004   #14
Mikkel deMib Svendsen
 
Mikkel deMib Svendsen's Avatar
 
Join Date: Jun 2004
Location: Copenhagen, Denmark
Posts: 1,576
Mikkel deMib Svendsen has much to be proud ofMikkel deMib Svendsen has much to be proud ofMikkel deMib Svendsen has much to be proud ofMikkel deMib Svendsen has much to be proud ofMikkel deMib Svendsen has much to be proud ofMikkel deMib Svendsen has much to be proud ofMikkel deMib Svendsen has much to be proud ofMikkel deMib Svendsen has much to be proud ofMikkel deMib Svendsen has much to be proud of
There are a lot fo reasons to detect either IP or agent names and change the content based on it. If you only check sites using Googlebot agent names you will mostly find sites that do agent identification for browser adjustments - not for SEO-cloaking.

Spoofing IP is not only difficult - it is highly illegal, I believe in most nations. So don't start on that just for de-cloaking purpose.
Mikkel deMib Svendsen is offline   Reply With Quote
Old 08-31-2004   #15
cline
Aderit Internet Marketing Consulting: www.Aderit.com
 
Join Date: Jun 2004
Location: Boston, MA
Posts: 146
cline has a spectacular aura aboutcline has a spectacular aura about
How about just comparing what G has cached vs. what you see?

Or is that too simple?
cline is offline   Reply With Quote
Old 09-01-2004   #16
rustybrick
 
rustybrick's Avatar
 
Join Date: Jun 2004
Location: New York, USA
Posts: 2,810
rustybrick has much to be proud ofrustybrick has much to be proud ofrustybrick has much to be proud ofrustybrick has much to be proud ofrustybrick has much to be proud ofrustybrick has much to be proud ofrustybrick has much to be proud ofrustybrick has much to be proud of
Most sites that cloak also use the noarchive request
rustybrick is offline   Reply With Quote
Old 09-01-2004   #17
mcanerin
 
mcanerin's Avatar
 
Join Date: Jun 2004
Location: Calgary, Alberta, Canada
Posts: 1,564
mcanerin has a reputation beyond reputemcanerin has a reputation beyond reputemcanerin has a reputation beyond reputemcanerin has a reputation beyond reputemcanerin has a reputation beyond reputemcanerin has a reputation beyond reputemcanerin has a reputation beyond reputemcanerin has a reputation beyond reputemcanerin has a reputation beyond reputemcanerin has a reputation beyond reputemcanerin has a reputation beyond repute
Quote:
Most sites that cloak also use the noarchive request
I find it amusing that cloakers assume that Google will obey the rules.

I find it astonishing that Google appears to actually obey them.

Ian
__________________
International SEO
mcanerin is offline   Reply With Quote
Old 09-01-2004   #18
Mikkel deMib Svendsen
 
Mikkel deMib Svendsen's Avatar
 
Join Date: Jun 2004
Location: Copenhagen, Denmark
Posts: 1,576
Mikkel deMib Svendsen has much to be proud ofMikkel deMib Svendsen has much to be proud ofMikkel deMib Svendsen has much to be proud ofMikkel deMib Svendsen has much to be proud ofMikkel deMib Svendsen has much to be proud ofMikkel deMib Svendsen has much to be proud ofMikkel deMib Svendsen has much to be proud ofMikkel deMib Svendsen has much to be proud ofMikkel deMib Svendsen has much to be proud of
Quote:
I find it astonishing that Google appears to actually obey them.
They don't. Only when and if they want (or by accident, I don't know). I believe I am propably not the only one that still remember 3 month after Google released the no archive tag and deleted all sites using it! Remember?

That even is one of the reasons I NEVER again listen to "good advise" from Google. I listen, yes, but then I do what I find right
Mikkel deMib Svendsen is offline   Reply With Quote
Old 09-01-2004   #19
seobook
I'm blogging this
 
Join Date: Jun 2004
Location: we are Penn State!
Posts: 1,943
seobook is a name known to allseobook is a name known to allseobook is a name known to allseobook is a name known to allseobook is a name known to allseobook is a name known to all
Quote:
Originally Posted by Mikkel deMib Svendsen
I believe I am propably not the only one that still remember 3 month after Google released the no archive tag and deleted all sites using it! Remember?
Wow, good deal for all
__________________
The SEO Book
seobook is offline   Reply With Quote
Old 09-01-2004   #20
Mikkel deMib Svendsen
 
Mikkel deMib Svendsen's Avatar
 
Join Date: Jun 2004
Location: Copenhagen, Denmark
Posts: 1,576
Mikkel deMib Svendsen has much to be proud ofMikkel deMib Svendsen has much to be proud ofMikkel deMib Svendsen has much to be proud ofMikkel deMib Svendsen has much to be proud ofMikkel deMib Svendsen has much to be proud ofMikkel deMib Svendsen has much to be proud ofMikkel deMib Svendsen has much to be proud ofMikkel deMib Svendsen has much to be proud ofMikkel deMib Svendsen has much to be proud of
Quote:
Wow, good deal for all
How? The majority of cloakers already knew about it so they where not hit. Most of the sites I saw dissapear was good sites that would have served the users well
Mikkel deMib Svendsen is offline   Reply With Quote
Reply


Currently Active Users Viewing This Thread: 1 (0 members and 1 guests)
 
Thread Tools

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

vB code is On
Smilies are On
[IMG] code is On
HTML code is Off