Sept. 2005 Google Index Update & Size Increase Coming?

[dannysullivan]

From Matt's blog:

Usually, what registers with an update to the webmaster community is when we update an algorithm (or its data), change our scoring algorithms, or switch over to a new piece of infrastructure.

Technically Update Gilligan is just backlink/PageRank data becoming visible once more, not a real update. There haven’t been any substantial algorithmic changes in our scoring in the last few days. I’m happy to try to give weather reports when we do our update scoring/algo data though.

If you go with update meaning an algorithm change -- and specifically an algorithm change that produces widespread notice and comment -- sure, this isn't an update.

But this isn't "everflux" or normal either. This isn't just backlink/PageRank data flowing in. This is likely billions of new pages coming in -- or perhaps a new way of counting existing pages. If these are new pages being added, that's update enough in my book, even if it doesn't meet the "traditional" definition. Moreover, you dump in a ton of new pages, that has a natural impact on rankings.

By the way, a search for the now brings back 8,640,000,000 matches, which exceeds the 8,168,684,336 claimed on the home page. And to put that really in prespective, Google has a sizable number of documents -- in excess of a billion -- that aren't in English and wouldn't have the word "the" on them.

I'll leave you with this:

http://www.google.com/search?q=dkfkdjkfjdk = zero matches
http://www.google.com/search?q=-dkfkdjkfjdk = 9,600,000,000 matches

The first query is for a term that doesn't exist on any page in the index, apparently.

The second query is for - that term, ie, show all the pages that don't have it. Since we know it doesn't exist, we should get a count of all pages.

Last week, you couldn't do a query like this. Couldn't at all. I know, because I tried and suggested to Google that if they made it possible, then anyone could find their exact index size in exactly the way you used to be able to do this on Northern Light when it was all into wanting to prove index size.

So fair to say, that index is in the 9.6 billion range. And even there, I suspect it might be higher.

[Marcia]

There may be more being revealed, but this is not an update.

[dannysullivan]

Well, we disagree

In the days before we had Google, we had search engines updating which often mean having a major size increase. Inktomi would update, Infoseek would do it and so on.

This has all the looks of a major size increase. It's screwing the counts of ton of people. I keep getting emails from people who are wondering why they suddenly have more pages listed. It is a major change going on to the Google index. I don't think anyone will deny that, regardless of what we want to label it.

The rankings may not be changing, but then again we've also had past updates -- real honest to goodness algo shifts -- that have gone largely unnoticed. So is it an update only when people notice ranking shifts? Or an update when something major happens but no one notices? Or when something major happens that doesn't change rankings but still gets noticed?

Whatever the case, whatever is going on IS being noticed by many people. And as I said, that's an update in my books and why I think it's worth warranting and noting it as an update, regardless of Matt saying it's not because it's just new blacklink and PageRank data coming on.

Moreover, that's clearly, clearly not the full story. If Matt had posted -- it's not an update, we've just done the regular backlink and PageRank data export and by the way, we added another 2-4-6-8 billion pages, I think more people might step up to say, "Hmm, feels like that's kind of update territory to me." But he can't say that.

He can't say that because Google is clearly wanting to finish getting its index growth up and out there and probably to coincide with some study it can put out to "prove" that it now really is bigger than Yahoo. So we sit and wait until that's done, people wondering why those counts are going up on Google, but no clear answer on Google other than "it's not an update."

Perhaps we ought to go back to Google Dance terminology. We're not having a dance right now. The results aren't radically shifting around. To have a dance, you have to have an update -- to have an update, you don't necessarily have to dance.

[PhilC]

I'm seeing 11,360,000 results for that -nonsesnse search. Maybe it will just keep going up for a while.

Frankly, I simply don't believe it. If Google had billions of different pages to add to the index, why hadn't they added them already? They certainly didn't just find them on the Web. I never believed their 8 billion figure, because when they doubled that from 4 billion, they also doubled the reported number of pages from one of my sites, which results in 2 - 2½ times the maximum possible from that site being reported. So I always assumed that they were counting many, if not all, pages in the index at least twice. Perhaps they are adding up the totals from their several indexes, even though the pages are represented in more than one index.

For them to suddenly increase from 8 billion to 11 billion (and rising) in one go looks more like juggling with numbers, as before, than a genuine increase in the number of pages they have.

As for what is and isn't an update, we've generally thought of a normal update as being to do with the algo, and a backlinks/PageRank update as being an update of a different kind. If they add a stack of new pages in one go, it would be another type of update again. When they periodically run filters over the index, it's yet another type of update. I suppose that any significant changes can be called updates.

[Rob]

I don't understand how this can NOT be an update.

I'm seeing clients whose page count has doubled or tripled this week alone. thats not backlink imports or PR data changing - its obviously an update. Not a Dance update as Danny puts it but still an update.

If it wasn't an update I wouldn't have a clients site jump from 430,000 to 780,000 to 1,670,000 indexed pages in the span of one week.

[PhilC]

Originally Posted by Rob

If it wasn't an update I wouldn't have a clients site jump from 430,000 to 780,000 to 1,670,000 indexed pages in the span of one week.

It sounds more like a manipulation of numbers by the minute

[Rob]

This is true, but if that's the case, and it does indeed involve a PR and BL update, then those "new" pages should count towards the overall incoming links and influence PR. therefore the site should change rankings which would then indicate an update, no?

It's like a catch 22...

[PhilC]

If it's just numbers, I wouldn't think it would make any difference.

[AussieWebmaster]

It has already added 8 pages of the nonsense terms.....

[DarkMatter]

I'm surprised no one has posted this in this thread yet. Matt Cutts says this is just a normal backlink/pagerank update, not an algorithmic change.

http://www.mattcutts.com/blog/whats-an-update/

[AussieWebmaster]

I is mentioned on here somewhere... just not in this thread.

[BroadProspect]

Ok, now I also see a huge SERPs changes on -cw: 216.239.37.99 -ex: 216.239.39.99

I am also seeing the realtime hits on my server (don't you just love the tail -f on the accessLog file and it is JUMPING !
let the music start !
/BP

[Rob]

So I was doing "the query" to check the size, but using different numbers - something totally at random and I noticed that the size doesn't change, but neither does the top listings. It doesn't seem to matter what types of random characters, fronted with a "-" I put I always seem to get the same results.

I wonder... Could these sites be the ones that Google considers the most relevant over all?

Currently I see:

#1 - www.alistapart.com
#2 - corporate.britannica.com/library/info/
#3 - http://www.firstgov.gov/Topics/Refer...ibraries.shtml

and so on. It doesn't matter what combination of random characters I always see the same sites.

Any thoughts?

[orion]

Originally Posted by dannysullivan

http://www.google.com/search?q=-dkfkdjkfjdk = 9,600,000,000

Good trick, Danny.

Same search in Yahoo returns nothing, but a search for the letter "a" returns

Results 1 - 10 of about 11,500,000,000 results


while in Google a search for "a" gives

8,000,000,000

According to the above numbers and reasoning:

1. could we say that in Google 1.6 billion pages do not contain the letter "a"?
2. could we say that Yahoo has an index of at least 11.5 B pages?

Both search engine results either can be trusted or can be rendered as suspicious, depends on who you ask.

Trying to estimate the size of a search engine from scores driven by queries is not a reliable approach and raises more questions than answers.

Orion

[GOTseo]

Well a site I manage was indexed 989 times then it surged to 10,4000 serps then 11,100 then back down to 974.

I was figureing out how google managed to spider the site for 11,000 products when there are only 182 products in the site.


Is this a possible flux that will drop the indexed pages after the next update?

[scubajared]

I have a client, a Realtor, who was listing #1-#3 for all of the major buildings in South Florida before Sunday. Today he has dropped down to the bottom of the first page or second page for most of terms that he was placing #1 for. The major term which we were going after Miami Real Estate actually moved up a few places which to me makes no sense. The sites that replaced him for the top positions were less relevant sites, sites that I never seen in Google and sites that almost have no content about the terms or back-links to those pages.

The only thing I can think of was that his blog had a lot of track-back spam that we did not catch. Could this of caused the drop? I don't understand how less searched terms dropped off which gave us over 250 differnet keyword searches in Google a day to less then 50.

Thanks for any info

Jared

[KevinSource]

Jared

Many people are seeing changes in Google as of this weekend. See this thread.

KevinSource

[wizardprincess]

this can be a major update as my many sites are drop from GOOgle SERP