MSN Powered by Inktomi?

[bethabernathy]

Hi There: I have been reading articles and posts in the forums and I am confused, is MSN powered by Inktomi or Yahoo! search? I realize that Yahoo purchased Inktomi, so there may some semantic issue related to the way people are stating MSN is powered by Yahoo, but I did read on one thread in this forum where Danny Sullivan stated:

"shades of what used to happen back in the old days when we still had an Inktomi and it powered other people."

Posted at:

http://forums.searchenginewatch.com/forum/showthread.php?threadid=62

I several clients who paid for Inclusion with Inktomi and if it is no longer then their contracts have been terminated before their terms end. Thanks for any information on this. -Beth

[David Wallace]

The last time I checked, MSN was still being powered by the Inktomi index. Of course, I wonder if the Inktomi index is really a version of the Yahoo index, just not as up to date. :confused:

[bethabernathy]

I am not sure if Inktomi is gone, but at Hotbot its' logo is still listing at the bottom of the search results.

[Dodger]

All I know is that to some extent Inktomi still exists. The Inktomi spider is definitely still crawling, but it is also crawling non-Inktomi inclusions that were never submitted. Why that is, I cannot say. But if this is actually the Slurp spider, then Yahoo should at least rename it.

[bethabernathy]

Shouldn't MSN disclose who provides their results?

[Dodger]

Shouldn't MSN disclose who provides their results?

Disclose to who? Us or the searchers?

You can go to Bruce Clay's Website and see who is supplying who with his Search Engine Relationship Chart (http://bruceclay.com/searchenginerelationshipchart.htm).

I do not think it makes a difference to the searchers, as long as the Overture results are clearly marked at MSN as being paid results.

[MakeMeTop]

>I several clients who paid for Inclusion with Inktomi and if it is no longer then their contracts have been terminated before their terms end.

Inktomi (as an index supplier) still remains - and will do until people who have paid for Inktomi PFI have had their contracts expire. These "Inktomi" results will still show in the results where the previous Inktomi data was used. Currently this includes Hotbot and MSN - though either may change. Confusingly, it does not include Yahoo. However, Yahoo Search! results also show in previous Inktomi partners.

[dannysullivan]

The official word from Yahoo is that there is no separate Inktomi index. There's only the Yahoo Search Technology index, and that's used by Yahoo and partners that Yahoo support directly through Yahoo, though the Yahoo-owned Overture subsidiary or the Yahoo-owned Inktomi operation.

That other thread talked about how despite their being one index, results at Yahoo may be much different than results at Yahoo's partners. This, according to Yahoo, is due to those partners perhaps using different ranking criteria for their indexes.

In reality, it looks like Yahoo is pushing out one version of a ranking algorithm combined with hitting only a subset of its total index to partners, while the full index and a different ranking algorithm is used by Yahoo itself.

You could call that smaller index "Inktomi," but I think that it's better to think of it as Yahoo and understand that Yahoo results may not always be the same as at Yahoo. That's important, because it suggests that you need to be in both Yahoo and Inktomi. There's no way to submit to just "Inktomi" any longer that I know of, not even through paid inclusion, because Inktomi no longer is supposed to exist.

It is odd there's still a spider out there calling itself Inktomi. I'd guess an oversight by Yahoo's part. As for why you see the Inktomi logo in places, there's still an "Inktomi" network being supported by various contractual obligations. And as Makemetop says correctly, those with paid inclusion contracts to be in "Inktomi" that still have time to run get put into these places and only these places if they've not bought Yahoo/Overture paid inclusion separately.

Life would have been much easier if Yahoo has simply said that anyone who'd bought Inktomi previously would have been grandfathered into the new Yahoo paid inclusion program.

FYI, guide to who powers whom from SEW is here (http://searchenginewatch.com/reports/article.php/2156401), recently updated.

[seobook]

I wonder why Lycos Insite is still selling Inktomi inclusion?
http://insite.lycos.com/inclusion/searchenginesubmit.asp

[dannysullivan]

I wonder why Lycos Insite is still selling Inktomi inclusion?

I'll see if I can get Lycos to comment -- my guesses:

1) They simply haven't changed the page (happens more than you'd realize)

2) They don't have a contract to sell inclusion on Yahoo, so selling Inktomi means what Yahoo euphemistically calls "the historic Inktomi network."

In particular, notice there's no per charge fee included with Inktomi -- another reason why this probably doesn't get you into Yahoo itself.

[Dodger]

It is odd there's still a spider out there calling itself Inktomi. I'd guess an oversight by Yahoo's part. As for why you see the Inktomi logo in places, there's still an "Inktomi" network being supported by various contractual obligations.

The WhoIs entries on those spiders have had the contact information changed to reply to Slurp-at-yahoo.com though. The answer section still reflects the Inktomi name. That is what is so confusing about it.

I'll see if I can get Lycos to comment -- my guesses:

2) They don't have a contract to sell inclusion on Yahoo, so selling Inktomi means what Yahoo euphemistically calls "the historic Inktomi network."

In particular, notice there's no per charge fee included with Inktomi -- another reason why this probably doesn't get you into Yahoo itself.

That is probably more like it, it is paid inclusion into the Lycos search results would be my guess.

And you gotta love that "historic Inktomi" line. The one I liked is when the Fast Enterprise crawler was going gonzo about a month and a half ago -- Fast was not under contract to crawl the pages, nor was Yahoo owning up to it being part of AllTheWeb's part of Fast. Tim Mayer used that same message when asked at WebmasterWorld. Nobody was owning up to this little bugger that I could see.

USCITY.NET logged over 22 gigabytes of data pulled by AlltheWeb's crawlers attempting to harvest USCITY.NET's member links. AlltheWeb's dysfunctional crawlers also generated over 12,000 errors on USCITY.NET servers scavenging for pages that don't exist. Repeated attempts to contact AlltheWeb and Overture about its questionable crawling yielded no response.

This all happened during the transition period when they were combining all of the Search products. My feeling is that something went unmonitored or they were unaware of the spider getting loose ... hehehe. But there has been no open acknowledgement of that.

[bethabernathy]

I ran the log files for a client and the spider is still named Inktomi:

Domain Name Hits Users Time of Last Access
inktomisearch.com 2 1 April, 06 2004 10:20:18 AM

Although that was in April. I'll check some newer logs.

The thing that was awful about the Yahoo paid inclusion was that I was lead to believe, some way or another, that the pages I included prior to the switch the pages would be grandfathered into the new Yahoo! index as well as into Alltheweb & MSN. I got the news just after the Google Florida and my clients were desparate to get some referrals, short of adwords, so we paid and got all their pages included. Then I was told about a month later that the pages would not be included in the Yahoo! index just MSN & Hotbot. Luckily I have very nice clients, but I went and created site maps for most of them and submitted the sitemaps to Yahoo!s new search engine and pleasantly to my surprise, most all the pages and more have been included in the Yahoo! index. It was a pretty frightening time. I think I have been complaining about this for a bit too long. -Beth

[bethabernathy]

I checked some log files for June 2004 and it still is calling the spider Inktomi search. This is the record:

Domain Name | Hits | Users | Bandwidth
inktomisearch.com | 90 | 46 | 5.93 megabytes

[TheotherTim]

I think you will find that the user agent of the spider has changed to Yahoo Slurp but the domain has remained Inktomisearch. This was not overlooked and has been on our list of things to migrate over the coming months.
The former FAST Crawler is still out there crawling in a limited capacity. It has not been crawling for web results for any of our production sites for several months. I think you will notice the user agent says "former web crawler" which identifies that fact. This is a fact that the uscity.net people failed to notice or realize before issuing their "press release".

[seobook]

Hi Tim
could you tell us what the Lycos Inktomi purchase gets us :)

[TheotherTim]

The Lycos Inktomi purchase is a way to buy distribution into the legacy Inktomi network. This excludes Yahoo. You may want to ask Lycos directly for more info.

[Dodger]

Thanks Tim. I kind of knew the Inktomi domain was Slurp -- today anyway.

Back during the USCity event their spider was probably the same one that I was seeing as well. This was one of them:

User-Agent: Yahoo-Verticalcrawler-FormerWebcrawler/3.9 crawler at trd dot overture dot com; http://www.alltheweb.com/help/webmaster/crawler
IP: 68.237.225.195 (pool-68-237-225-195.ny325.east.verizon.net)

This was brought up at WebmasterWorld (http://www.webmasterworld.com/forum35/2107.htm) at the time in which you stated:

"This crawler has nothing to do with atW anymore other than it was the crawling technology that formerly discovered URLs for the FAST/atW Engine. This crawler could be crawling for a variety of search products or projects at Yahoo."

I tried contacting FAST and got no response from them. But while I was there, it was interesting to note (at that time) that they were still fulfilling outstanding obligatory contracts per their homepage. That could mean just about anything, I know -- but it made things even more confusing.

It was also at that time that I noticed that they dropped AllTheWeb as one of their Express partners (as they call them), but added Overture to the list. This confused me even more. (I have not gone back to see if they are still on the list).

As you can see there was a spider out there doing work for somebody or someone...and it was really going gonzo too. Two days I methodically watched this thing start at the top and work it's way down. It was actually pretty good at weaving thru a forum's links. I was actually pretty impressed. But then it started back at the top again, and that is when we pulled the plug.

It was following the FAST rules of engagement, i.e. one page request per minute. It was definitely a FAST spider, but it was also a spider for hire too.

[trellian]

>I several clients who paid for Inclusion with Inktomi and if it is no longer then their contracts have been terminated before their terms end.
Inktomi (as an index supplier) still remains - and will do until people who have paid for Inktomi PFI have had their contracts expire. These "Inktomi" results will still show in the results where the previous Inktomi data was used. Currently this includes Hotbot and MSN - though either may change. Confusingly, it does not include Yahoo. However, Yahoo Search! results also show in previous Inktomi partners.

Hello Barry, as you mayd be aware, the terms of Inktomi services were never tied in specifically to any search engine. If MSN drops Inktomi/Yahoo today, the Inktomi service that you have paid for will still be honored by Inktomi and the remaining portal partners. Inktomi have always made sure that the service offered, was for Inktomi inclusion, not MSN or any other specific portal inclusion program, hence as long as one portal displays Inktomi results for the full term, the service terms will be fulfilled.

Cheers
David

[Winklerb]

One of my client's was completely dropped from MSN's search engine (after 4 years of reasonably steady traffic growth), apparently with the new "googleesque" makeover.
Is there any recourse other than the pricey Overture listing service to reestablish the MSN hits?

[trellian]

After speaking with someone at MSN afew days back, the news is: "we are considering all our options". As soon as we know what PFI program (if any) they choose to go with I am sure it will be one that looks at relevancy above all other factors.

Cheers
David