spider simulator for dev sites

[rakeshmp]

hey, does anyone know where i can get the code for a spider simulator to test on dev sites? thanks.

[Marcia]

Here you go, take your pick:

http://www.google.com/search?hl=en&r...er+sim ulator

I doubt if they're giving away the actual code, but some may allow you to use a link or form on a web page. Another option is to download and use Lynx browser, which is an all text user agent, which is what search spiders are, and will emulate how they'll see pages.

[beu]

I also like looking at the cached text version of search results because sometimes you can glean information not reported by spider simulators.

[Marcia]

Originally Posted by beu

I also like looking at the cached text version of search results because sometimes you can glean information not reported by spider simulators.

True, and by looking at the snippets and how they're highlighted as well.

Here's just what you're looking for, an online tool and capability suggested for viewing pages in a production environment:

http://www.delorie.com/web/lynxview.html