How Do I Spot Cloaked Sites?

[rustybrick]

Forget the debate about cloaking, I am a bit tired of that anyway. How does one detect some of the cloaking going on around the Web. Follow these instructions:

(1) Download the Firefox Browser
(2) Install it
(3) Download the User Agent Switcher for Firefox/Mozilla while using firefox
(4) Restart the browser
(5) Under Tools --> User Agent Switcher --> Options --> Options (that will open a dialog box)
(6) Click Add Under User Agents section
(7) In the description add "Googlebot" and in the user agent add "Googlebot/2.1 (+http://www.googlebot.com/bot.html)"
(8) Repeat this process for all the spiders you want to test. Updated comprehensive list of user agents.
(9) Under Tools --> User Agent Switcher --> select the user agent
(10) Then navigate to the pages that you want to test for cloaking.

Hope this helps some people be Googlebot.

[Nick W]

Not bad Rusty.

Unfortunately that's only going to catch the incompetent cloakers. Most cloakers wouldnt give a stuff about UA's, they care about IP's

Shame firefox is so 'different' to Moz, some of the stuff is really round the wrong way, i just cant get used to it...

Nick

[mcanerin]

Unfortunately, although it's easy to spoof an IP one way, it's very hard to spoof an IP and actually get a response back (the response would go to the spoofed IP, not yours).

There are ways around this but it's pretty complex, and usually not worth it.

I wonder how often (if ever) Google employees go home with a bunch of websites and visit them from there That would be really funny. Heck, you could even have a separate line/ISP coming into Googleplex just for surfing the web on non-googlebot IP's. Would be easy if they wanted to. Could even write a script to automatically compare a site that had been flagged.

That would be lots of fun, too.....

Yes, I'm evil.....

Ian

[hiero]

Pretty cool Barry......anyone have a url to test?

[rustybrick]

Originally Posted by Nick W

Not bad Rusty.

Unfortunately that's only going to catch the incompetent cloakers. Most cloakers wouldnt give a stuff about UA's, they care about IP's

Shame firefox is so 'different' to Moz, some of the stuff is really round the wrong way, i just cant get used to it...

Nick

Nick, 100% right. But its funny to see those people who cloak without the IP delivery methods. Of course mcanerin idea would work, but you would need some powerful software thats updated frequently to find these sites that way. And then, if your doing that, you can then easily cloak yourself.

[hiero]

Here is a link to some cloaking examples:
http://fantomaster.com/fafaqcloak4.html

Using their first example:
Now if I'm using Firefox with the user-agent set to googlebot I should be seeing the bot page, but I'm not.

So I can only assume that the cgi script doesn't include googlebot 2.1 as an agent, right?

[rustybrick]

Um, well no. Fantomaster's products are a bit more sophisticated then simply changing the user agent. They use IP Delivery. They know Googlebot's IP addresses and based on that info, they redirect the bot to a different page. What Nick said above in post # 2.

[NFFC]

>they redirect the bot to a different page

I think the word redirect isn't quite right.

It seems to me that they provide a customised experience according to user expectations.

[hiero]

Got it, thanks.

Then what real value do you see the user agent switcher for Firefox having? Is it a worthwhile add on?

[Nick W]

It would help with those dumb arsed sites that insist you get IE b4 u can view their crappy AOL like pages.......

...as if you'd want to eh?

Nick

[rustybrick]

There are many sites that deploy user agent based cloaking only. Why? I guess because its easier.

[hiero]

So if I write a javascript routine that looks at the user agent and returns a page for that user agent that would be cloaking also. hmmmm.....weird.

[seomike]

Go to webmasterworld.com if you have a user agent with a search engine bot name you'll see meta tags otherwise you won't

[Mikkel deMib Svendsen]

There are a lot fo reasons to detect either IP or agent names and change the content based on it. If you only check sites using Googlebot agent names you will mostly find sites that do agent identification for browser adjustments - not for SEO-cloaking.

Spoofing IP is not only difficult - it is highly illegal, I believe in most nations. So don't start on that just for de-cloaking purpose.

[cline]

How about just comparing what G has cached vs. what you see?

Or is that too simple?

[rustybrick]

Most sites that cloak also use the noarchive request

[mcanerin]

Most sites that cloak also use the noarchive request

I find it amusing that cloakers assume that Google will obey the rules.

I find it astonishing that Google appears to actually obey them.

Ian

[Mikkel deMib Svendsen]

I find it astonishing that Google appears to actually obey them.

They don't. Only when and if they want (or by accident, I don't know). I believe I am propably not the only one that still remember 3 month after Google released the no archive tag and deleted all sites using it! Remember?

That even is one of the reasons I NEVER again listen to "good advise" from Google. I listen, yes, but then I do what I find right

[seobook]

Originally Posted by Mikkel deMib Svendsen

I believe I am propably not the only one that still remember 3 month after Google released the no archive tag and deleted all sites using it! Remember?

Wow, good deal for all

[Mikkel deMib Svendsen]

Wow, good deal for all

How? The majority of cloakers already knew about it so they where not hit. Most of the sites I saw dissapear was good sites that would have served the users well