MSNBot Beats Googlebot

[Dave Hawley]

I have noticed that for the last few days msnbot has been spidering our site even more than googlebot. This is the first time ever, for us, that this has occured.

Perhaps MSN will soon announce a >8 billion page database?

[goodroi]

MSN has been hungrier than Google for several months. As for Google's count of 8 billion, that is not accurate. The big 3 search engines are regularly adding pages to the index and rarely change their numbers.

[TinkyWinky]

To right - no way there's 8 billion real pages in there:

- there's about 900 (at last count) links to pages I had excluded the Googlebot from

- there's a dup site where the .com version acts as a separate site from .co.uk (yes developer didn't do that quite right!!!)

- there's a whole load of old cach'd pages from a site I deleted about 2 years ago

Nice one Google - seems they are hell bent on being the biggest - soon see whether they are the best.

MSN on the other hand is not perfect - but the bot seems to come through everyday. On one new domains site I put live 30 content pages yesterday - they are all included already. Now that's a search engine! :)

[Marcia]

>>MSN has been hungrier than Google for several months.

Does anyone have any suspicion that clickthroughs in the reglar MSN serps may trigger visits from the bot?

[Nacho]

MSNBot Beats Googlebot

1) Not in my books. Googlebot crawls deeper, faster and comes back more often than MSNBot, but. . .

2) Crawling is very different than crawling+procesing+analyzing+indexing+making it available for search, so . . .

How much a crawler hits your site doest really say much, in my opinion.

[Dave Hawley]

MSN has been hungrier than Google for several months.I take it you mean on your site(s).

Not in my books. Googlebot crawls deeper, faster and comes back more often than MSNBot, Yes, this exactly what I always see, however, yesterday was an exception. In fact, Google has been so hungry on our site the last few month I had to email them and say slooooowwww down please. They were chewing up 30GB a month for 3 months straight! It took 3 emails over about 3 weeks but they did slow down.

I tend to believe that Google now have 8 billion pages in their database. However, I think the way msnbot is going, they too may soon announce a similar size....or perhaps more.

I'm quite excited about MSN Beta going live. I have good rankings with them (hope they don't tweak too much) and to start getting a higher percentage from them would stop us being so reliant on Google, which I have never liked.

[goodroi]

I take it you mean on your site(s).
I've been tracking my network of sites and compiling data with other people who run networks of sites. All spiders seem to be hungrier than they used to be. MSN when it first came out was intense and would try to crawl everything at once but was not as consistent and as Google. They've been becoming more regular. It doesn't realy matter if the spiders are pounding your site. What matters is if they rank your site :) or if they overburden your server :(.

If you want to increase your chances of being spidered, get links. If you want to increase the frequency of return spidering, change your content very often (more links wouldn't hut either). If you want to slowdown a spider you can try the crawl-delay command in robots.txt or email the search engine.

[Vural Cifci]

I think that was a really correct observation. All spider became badder and meaner. :)
But I cannot wait to see the end results when things start becoming more stabilized and who has more relevant, quality resluts as well as quantity of the results.

[Carlos Chacón]

I think that was a really correct observation. All spider became badder and meaner. :)
But I cannot wait to see the end results when things start becoming more stabilized and who has more relevant, quality resluts as well as quantity of the results.

I am agreeing. In my case MSN represent not the big deal -talking about traffic or customers...although my site is listed.
And, between quantity and quality?
I prefer quality ;)

[Dave Hawley]

I'll take what pumps up the bottom line the most ;)

[rmouser]

One thing for sure, MSN beta us currently showing new links faster than any other search engine.

I have a new site where I added 15 links in October and 150 links in November, so far Google shows 24 of them, Yahoo shows 60, and MSN beta shows 164.

MSN has a different set of priorities than the other search engines (at least for right now).

Richard

[rmouser]

Forgot to mention, don't trust the number of links shown by MSN beta on the first few results pages of a link:www.domain.com search, they showed over 500 for my new site...don't I wish! :)

[Dave Hawley]

Are you saying the Link command, on MSN, was showing links to the page that didn't exist?

[powerofeyes]

Crawling is very different than crawling+procesing+analyzing+indexing+making it available for search, so . . .

How much a crawler hits your site doest really say much, in my opinion.

We see similiar things though MSNBOT hits some of our sites badly they dont seem to include those pages in their index, We saw some pretty active activity in some of our sites from msnbot but they didnt include the pages in the index they fetched from our sites,

In these things I agree googlebot is more superior, they crawl deeper as well as they include all the pages crawled in their main index,

[rmouser]

Are you saying the Link command, on MSN, was showing links to the page that didn't exist?

Dave,

Sorry I was not clear, it is only the total number of links (shown at the top of the first few results pages) that is inflated, once I browse forward through a few pages of results, the number becomes realistic.

I did not see any links that were unfamiliar...only the total count (e.g., 1-10 of 543, where 543 should be ~164).

Richard

[Dave Hawley]

Cheers Richard, I understand now and have seen the same thing.