Search Engine Watch
SEO News

Go Back   Search Engine Watch Forums > Search Engines & Directories > Yahoo! > Yahoo Web Search
FAQ Members List Calendar Forum Search Today's Posts Mark Forums Read

Reply
 
Thread Tools
Old 01-16-2007   #1
Marcia
 
Marcia's Avatar
 
Join Date: Jun 2004
Location: Los Angeles, CA
Posts: 5,476
Marcia has a reputation beyond reputeMarcia has a reputation beyond reputeMarcia has a reputation beyond reputeMarcia has a reputation beyond reputeMarcia has a reputation beyond reputeMarcia has a reputation beyond reputeMarcia has a reputation beyond reputeMarcia has a reputation beyond reputeMarcia has a reputation beyond reputeMarcia has a reputation beyond reputeMarcia has a reputation beyond repute
Site getting hammered by crawler

It keeps showing this

Quote:
Host: 72.30.216.22
/suspended.page/
Http Code: 404 Date: Jan 17 01:12:31 Http Version: HTTP/1.0 Size in Bytes: -
Referer: -
Agent: Mozilla/5.0 (compatible; Yahoo! Slurp; http://help.yahoo.com/help/us/ysearch/slurp)

Host: 74.6.67.78
/suspended.page/
Http Code: 404 Date: Jan 17 01:11:24 Http Version: HTTP/1.0 Size in Bytes: -
Referer: -
Agent: Mozilla/5.0 (compatible; Yahoo! Slurp; http://help.yahoo.com/help/us/ysearch/slurp)

Host: 74.6.74.155
/suspended.page/
Http Code: 404 Date: Jan 17 01:08:45 Http Version: HTTP/1.0 Size in Bytes: -
Referer: -
Agent: Mozilla/5.0 (compatible; Yahoo! Slurp; http://help.yahoo.com/help/us/ysearch/slurp)

Host: 74.6.71.43
/suspended.page/
...tens of thousands of times in a row.

Has anyone seen this happen? Is this a crawling problem or a server misconfiguration of some kind all of a sudden?
Marcia is offline   Reply With Quote
Old 01-18-2007   #2
vicyankees
Unknown User
 
Join Date: Jun 2005
Location: Unknown
Posts: 65
vicyankees is on a distinguished road
Do you have something in your basewebpage that a server error page also pulls from? looks like the spider hit an error and fell into a continuous cycle that would probably ultimately crash the server.
vicyankees is offline   Reply With Quote
Old 01-18-2007   #3
evilgreenmonkey
 
evilgreenmonkey's Avatar
 
Join Date: Feb 2006
Location: London, UK
Posts: 703
evilgreenmonkey has much to be proud ofevilgreenmonkey has much to be proud ofevilgreenmonkey has much to be proud ofevilgreenmonkey has much to be proud ofevilgreenmonkey has much to be proud ofevilgreenmonkey has much to be proud ofevilgreenmonkey has much to be proud ofevilgreenmonkey has much to be proud ofevilgreenmonkey has much to be proud of
Hi Marcia,

/suspended.page/ is a URL mainly used by the CPanel control panel, appearing when a site exceeds its bandwidth limit or is suspended by the web host. When the suspension is activated, all URLs for that domain will redirect (302 I think) to this page and therefore sometimes get included in the search engine's index.

If I was to hazard a guess, I would say that the site in question was suspended during a Slurp crawl and every URL in their index was visited and redirected to this URL.

I would counteract this by sticking the following in your robots.txt:

User-agent: *
Disallow: /suspended.page/

And then plea with your hosting provider to add the following to the top of the index.php file in the suspended.page folder:
PHP Code:
<?php
header
("HTTP/1.0 503 Service Unavailable");
?>
This should tell all the spiders to not index the folder and that the site is currently down for maintenance (rather then index a default suspended page).

If you have a CPanel reseller account (i.e. use WHM and can create new hosting accounts), you should be able to edit the contents of suspended.page yourself, as it's usually located in the root folder of your main hosting account.



Rob
evilgreenmonkey is offline   Reply With Quote
Reply


Currently Active Users Viewing This Thread: 1 (0 members and 1 guests)
 
Thread Tools

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

vB code is On
Smilies are On
[IMG] code is On
HTML code is Off