Sept. 2005 Google Index Update & Size Increase Coming?

[shor]

I am observing a large increase in pages indexed across our network, competitors and everyone inbetween.

Sample (5 sites):
Datacenters: 72.14.207.104, 64.233.187.104, 64.233.161.104

Date
5/09/2005 17:11 1,870,000 1,730,000 349,000 544,000 317,000
7/09/2005 13:00 4,910,000 4,660,000 828,000 1,512,000 864,000

Yes, one of the above sites went from under 2M to ~5M indexed pages. Some other datacenter blocks are showing 'old' index sizes. Anyone else seeing these changes?

[Mac]

Backlinks are being updated too.

[jan]

Google Directory was updated as well.

[inlogicalbearer]

Google have put back the most used stopwords to 8 billion. This normally precede an update in the index size number. You can compare the digit in my blog.

Moderator Note: See http://inlogicalbearer.blogspot.com/...-billions.html but more importantly http://inlogicalbearer.blogspot.com/...at-google.html to understand how the counts for queries on stop words like "the" can change.

[vicyankees]

My site has been increasing as of late but the past few days this number has been consistent among all data centers. A week ago we were at 1.5MM and today we are at 2.65MM.

[Rob]

I have noticed 2 sites with huge updates:

1 went from 340,000 to 710,000 indexed pages over the weekend, the other went from 1.2 million to 4.2 million pages indexed.

Guess Google wants to say "now ours is bigger"

[vicyankees]

Let the big boys fight - yahoo has had a steady 10-20k increase per day as well. When is MSN going to join in on the fun??

[dannysullivan]

Yep, Google About To Increase Index Size? over on the blog from me on Monday had some data centers a reader sent showing that a size increase looked likely. Guess it's coming.

[mtb167]

Google has picked up about 100 links coming from within my site to a particular page, but it is yet to pick up any links coming from external sources.

What's going on with that?

[dannysullivan]

Also, we have a related thread over here: Google Datacenters Backlink Update. I thought about merging that with this one but thought to keep them separate for now. DaveN may do that down the line, if he thinks it makes sense.

We also don't name the updates over here, leaving that to WebmasterWorld as their tradition. But if I were to name it, I'd call it something like Jedi. Why?

Well, Google's long been the biggest search engine based on self-reported counts. Then last month, Yahoo trumps them with 19 billion. If it were the Star Wars saga, that would have been "Empire Strikes Back." Now Google may be having its Revenge Of The Jedi moment.

Of course, I'm not saying Yahoo's the Dark Side! And in fact, perhaps Yahoo's index update would be more "Episode IV: A New Hope" and Google is more the Empire Striking Back. Update Vader?

[ukgimp]

40M to 135M for one serp i look at

No change in traffic, me thinks they are trying to show how big their index is with respect to Yahoo.

[vicyankees]

is there a tool or a method to determine which pages Google/Yahoo are including in their index? i'm using the site:xxx.com inurl:directory as a means of breaking it up as best as possible but when i do this i still fall wayyyyyyyy short of Google's number for total saturation. Any ideas?

[PhilC]

It would really be funny if Google suddenly upped the claimed number of pages in their index by a very large amount. It would be funny to us, and to the press I imagine, but I'm sure it would be believed by the general Web population, and they are the people who Google need to impress.

I'm still convinced that Google's main index is around the 4 billion mark.

[DaveN]

yep agree with danny this is UPDATE JEDI..

seeing increase in :

a) Internal pages from sites increased
b) more results been returned for keyword pharses
c) more lesser know TLD's appearing in the serps
e) more backlinks shown..

gut feeling, Google was getting ready to increase the Index size BEFORE SES San Jose.. Yahoo announce they increase.. google have 2 options say we have Aswell ... say nothing and increase their index at a later date..

I think it was the later.

going to run a quick test .. results to follow

[vicyankees]

I'm now up to 4MM pages on most datacenters...

[I, Brian]

Update Bloat - Google simply seem to have doubled the number of indexed pages on a site. Are they now simply counting www and non-www pages as separate?

Tongue in cheek perhaps, but as I reported on my blog (gratuitous self-link avoided), you can check the new datacenter figures for yourselves against when they first really kicked in with "results crowding" in December last year.

I see a third fewer results returned for "web" and "computers", half as many documents claimed exist containing the word "casino" and only one-third as many documents containing the word "porn".

The fewer results returned, in my opinion, is the real eye-opener - it seems that the larger the index Google claims to have, the fewer results they are prepared to return on a search.

[morepro]

Based on Googleguy's latest post today, it appears that everyone is jumping the gun a little bit...

http://www.mattcutts.com/blog/whats-an-update/

[samd]

http://www.google.co.uk/search?hl=en...e+Search&meta=

Searching for "and" gives more results than are (according to Google) in the index.

[samd]

Also:

Moving through the 8 Billion "the" results you can only get up to 1000.

http://www.google.co.uk/search?q=the...rt=100000&sa=N

The message displayed is:

"Sorry, Google does not serve more than 1000 results for any query. (You asked for results starting from 100000.)"

[PhilC]

Google never shows more than 1000 results for any search. It's always been like that.