Rob
05-26-2006, 02:26 PM
Hi
A few months ago I noticed that Googlebot was requesting pages on various sites that didn't exist. While they were similar to other pages on the site, they did not exist, nor had ever existed.
Well just recently I heard from one of the sites in question that Slurp has taken on this type of behavior.
What it's doing is requesting pages that do not exist yet are similar to pages that do exist.
For example, if a site is geographic and has pages about Austin, Houston and Dallas the spider seems to be requesting a page for Fort Worth even though that page doesn't exist. It seems to understand the page naming conventions and then modifies them looking for other pages that may exist but could be orphaned perhaps?
In any case I was wondering if anyone else is experiencing this sort of phenomenon?
Thanks
A few months ago I noticed that Googlebot was requesting pages on various sites that didn't exist. While they were similar to other pages on the site, they did not exist, nor had ever existed.
Well just recently I heard from one of the sites in question that Slurp has taken on this type of behavior.
What it's doing is requesting pages that do not exist yet are similar to pages that do exist.
For example, if a site is geographic and has pages about Austin, Houston and Dallas the spider seems to be requesting a page for Fort Worth even though that page doesn't exist. It seems to understand the page naming conventions and then modifies them looking for other pages that may exist but could be orphaned perhaps?
In any case I was wondering if anyone else is experiencing this sort of phenomenon?
Thanks