Should I be blocking the "Yahoo Adcrawler" in my robots.txt file or maybe only blocking specific paths?
I just see that the crawler keeps coming back and visiting the site hundreds and hundreds of times per day and hitting pages that have not been alive for over a year...
It seems that if this bot was set up to crawl ads and determine relevancy, that it would only be looking at ads that are currently in the account... but that is not the case... I have also seen charges and reporting for terms and ads that have been removed and from what I can tell, the only thing hitting or using that URL is the adcrawler.
Has anyone else been experiencing this? I even went so far as to create a new account and transfer anything over that was live, because there is a bug in their system that defaults an ad to content and with an old URL if, for some reason the ad no longer qualifies due to quality or other reason.... This means that if you create and ad, push it live, then make changes to it.... they can revert the changes without giving you access to it and publishing it with the old info, all with no ability to modify or disable it....
In case anyone is not aware of the crawler, here is a link...
Linky
What do you think?
01-20-2009
Similar Threads
