PDA

View Full Version : What on earth is Yahoo doing here?

Tooooons
10-20-2005, 03:03 PM
This crawler does not identify itself as Yahoo. It is accessing robots.txt 2 or 3 times for every other file it pulls from our site (which appears to be only graphics).

The IP traces to a "shop" URL in the Yahoo domain, which leads me to think Yahoo is simply grabbing all of our images for its shopping search.

Anyone with any conclusions? I find this kind of "indexing" ridiculous. Im fact, I don't even think this is indexing. Yahoo is just pulling stuff off our site for its own use other than getting us results in their search. Not so great a move. This has been happening all morning, and the frequency is impressive.

Here's from our log:


207.126.224.12 - - [20/Oct/2005:12:55:10 -0600] "GET /fredjason-score.gif HTTP/1.1" 200 10388 "-" "libwww-perl/5.803"
207.126.224.12 - - [20/Oct/2005:12:55:10 -0600] "GET /robots.txt HTTP/1.1" 200 194 "-" "libwww-perl/5.803"
207.126.224.12 - - [20/Oct/2005:12:55:10 -0600] "GET /robots.txt HTTP/1.1" 200 194 "-" "libwww-perl/5.803"
207.126.224.12 - - [20/Oct/2005:12:55:11 -0600] "GET /fredjason-score.gif HTTP/1.1" 200 10388 "-" "libwww-perl/5.803"
207.126.224.12 - - [20/Oct/2005:12:55:11 -0600] "GET /robots.txt HTTP/1.1" 200 194 "-" "libwww-perl/5.803"
207.126.224.12 - - [20/Oct/2005:12:55:11 -0600] "GET /robots.txt HTTP/1.1" 200 194 "-" "libwww-perl/5.803"
207.126.224.12 - - [20/Oct/2005:12:55:11 -0600] "GET /fronnection.gif HTTP/1.1" 200 11748 "-" "libwww-perl/5.803"
207.126.224.12 - - [20/Oct/2005:12:55:12 -0600] "GET /robots.txt HTTP/1.1" 200 194 "-" "libwww-perl/5.803"
207.126.224.12 - - [20/Oct/2005:12:55:12 -0600] "GET /robots.txt HTTP/1.1" 200 194 "-" "libwww-perl/5.803"
207.126.224.12 - - [20/Oct/2005:12:55:12 -0600] "GET /fronnection.gif HTTP/1.1" 200 11748 "-" "libwww-perl/5.803"
207.126.224.12 - - [20/Oct/2005:12:55:13 -0600] "GET /robots.txt HTTP/1.1" 200 194 "-" "libwww-perl/5.803"
207.126.224.12 - - [20/Oct/2005:12:55:13 -0600] "GET /robots.txt HTTP/1.1" 200 194 "-" "libwww-perl/5.803"
207.126.224.12 - - [20/Oct/2005:12:55:13 -0600] "GET /fronnection.gif HTTP/1.1" 200 11748 "-" "libwww-perl/5.803"
207.126.224.12 - - [20/Oct/2005:12:55:14 -0600] "GET /robots.txt HTTP/1.1" 200 194 "-" "libwww-perl/5.803"
207.126.224.12 - - [20/Oct/2005:12:55:14 -0600] "GET /robots.txt HTTP/1.1" 200 194 "-" "libwww-perl/5.803"
207.126.224.12 - - [20/Oct/2005:12:55:14 -0600] "GET /fri.gif HTTP/1.1" 200 13200 "-" "libwww-perl/5.803"
207.126.224.12 - - [20/Oct/2005:12:55:15 -0600] "GET /robots.txt HTTP/1.1" 200 194 "-" "libwww-perl/5.803"
207.126.224.12 - - [20/Oct/2005:12:55:15 -0600] "GET /robots.txt HTTP/1.1" 200 194 "-" "libwww-perl/5.803"
207.126.224.12 - - [20/Oct/2005:12:55:16 -0600] "GET /fri.gif HTTP/1.1" 200 13200 "-" "libwww-perl/5.803"
207.126.224.12 - - [20/Oct/2005:12:55:16 -0600] "GET /robots.txt HTTP/1.1" 200 194 "-" "libwww-perl/5.803"
207.126.224.12 - - [20/Oct/2005:12:55:17 -0600] "GET /robots.txt HTTP/1.1" 200 194 "-" "libwww-perl/5.803"
207.126.224.12 - - [20/Oct/2005:12:55:17 -0600] "GET /fri.gif HTTP/1.1" 200 13200 "-" "libwww-perl/5.803"
207.126.224.12 - - [20/Oct/2005:12:55:18 -0600] "GET /robots.txt HTTP/1.1" 200 194 "-" "libwww-perl/5.803"
207.126.224.12 - - [20/Oct/2005:12:55:18 -0600] "GET /robots.txt HTTP/1.1" 200 194 "-" "libwww-perl/5.803"
207.126.224.12 - - [20/Oct/2005:12:55:18 -0600] "GET /frid.gif HTTP/1.1" 200 10081 "-" "libwww-perl/5.803"
207.126.224.12 - - [20/Oct/2005:12:55:19 -0600] "GET /robots.txt HTTP/1.1" 200 194 "-" "libwww-perl/5.803"
207.126.224.12 - - [20/Oct/2005:12:55:19 -0600] "GET /robots.txt HTTP/1.1" 200 194 "-" "libwww-perl/5.803"
207.126.224.12 - - [20/Oct/2005:12:55:19 -0600] "GET /frilights.gif HTTP/1.1" 200 11271 "-" "libwww-perl/5.803"
207.126.224.12 - - [20/Oct/2005:12:55:19 -0600] "GET /robots.txt HTTP/1.1" 200 194 "-" "libwww-perl/5.803"
207.126.224.12 - - [20/Oct/2005:12:55:20 -0600] "GET /robots.txt HTTP/1.1" 200 194 "-" "libwww-perl/5.803"
207.126.224.12 - - [20/Oct/2005:12:55:20 -0600] "GET /frilights.gif HTTP/1.1" 200 11271 "-" "libwww-perl/5.803"
bhartzer
10-20-2005, 03:13 PM
It's probably being used for their shopping engine. Have you checked to see if they're listing you in Yahoo! Shopping?
Tooooons
10-20-2005, 03:18 PM
It looks like they are. This checkup alerted me to a bunch of our products incorrectly listed with a price of $1!

We've been uploading our catalog to Froogle since December 2002, but nothing to Yahoo as they didn't have a free upload. Do they now? I'll go look around. I would appreciate suggestions, thank you.