Join Date: Mar 2005
Should I be blocking the "Yahoo Adcrawler" in my robots.txt file or maybe only blocking specific paths?
I just see that the crawler keeps coming back and visiting the site hundreds and hundreds of times per day and hitting pages that have not been alive for over a year...
It seems that if this bot was set up to crawl ads and determine relevancy, that it would only be looking at ads that are currently in the account... but that is not the case... I have also seen charges and reporting for terms and ads that have been removed and from what I can tell, the only thing hitting or using that URL is the adcrawler.
Has anyone else been experiencing this? I even went so far as to create a new account and transfer anything over that was live, because there is a bug in their system that defaults an ad to content and with an old URL if, for some reason the ad no longer qualifies due to quality or other reason.... This means that if you create and ad, push it live, then make changes to it.... they can revert the changes without giving you access to it and publishing it with the old info, all with no ability to modify or disable it....
In case anyone is not aware of the crawler, here is a link... Linky
What do you think?
|Currently Active Users Viewing This Thread: 1 (0 members and 1 guests)|
|Thread||Thread Starter||Forum||Replies||Last Post|
|Yahoo Small Business Hosting Train Wreck||ackmnmn||Other Yahoo! Issues||3||04-19-2007 08:59 AM|
|How to Contact Management at Yahoo||eslincanada||Other Yahoo! Issues||0||03-14-2007 12:49 PM|
|56 Directories To Submit To||bobmutch||Other Search Engines & Directories||105||07-19-2005 06:29 AM|
|Keynote with Jerry Yang - SES NYC 05||rustybrick||SEM Related Organizations & Events||0||03-01-2005 09:42 AM|