BigDaddy, Missing Pages In Google & Is The Big G Out Of Space

[Relevancy]

MODERATOR NOTE: I've merged posts from Where are all the SEW Moderators? relevant to the BigDaddy issues some are having into this thread.

No you are right Mods aren’t the only ones with answers that is why I had to go to webmasterworld to get into a in-depth discussion about this topic

http://www.webmasterworld.com/forum30/34055.htm

No one is really talking about it here. Few scattered threads here for such a huge issue.. I don't get it.

[JohnW]

There is a lot of discussion on some forums, but it doesn't go anywhere.

IMO what is being seen is just unplanned after-effects of BD. G rolled out some new technology to fix some things, and in the process they seem to have broken some other things. There isn't much to say about it other than it’s broken - it is what it is and will be over when it's over.

It does seem to be settling down, one DC at a time.

[Marcia]

in-depth discussion about this topic

Have there been any in-depth comments and observations about how Google's *possible* use of historical data is related to some of the problems we're seeing right now? If so, please point me to the specific posts; I'd love to see what others may be spotting with this.

As for myself, I'm taking an in-depth look at several sites under 100 pages related to that very thing, as well as the handling of 301 and 302 redirects. I'd also like to be pointed to any specific posts related to this in those threads. I've read through and haven't seen any on either issue that might relate to what I'm seeing.

And Yahoo too, on penalties and redirect issues - good time to look at both engines and connect the dots with any issues that may cause related problems on both, or problems rooted in the same factors, even if they have different effects.

[JohnW]

>I've read through and haven't seen any on either issue that might relate to what I'm seeing.

Exactly. But I have noticed things coming back to normal (or better) on certain DCs. It also looks like some sites are emerging for the first time on these same DCs.

Take a look at results on 64.233.167.104 and 72.14.207.104

[dannysullivan]

No one is really talking about it here. Few scattered threads here for such a huge issue.. I don't get it.

My experience is different audiences; different activiities.

WMW, in my view, is heavily affiliate/adsensers/non-content site owners. IE, people who aren't running web sites with a lot of content but pushing people to sites with content and making money along the way.

DISCLAIMER!!!! The above is a gross overgeneralization, OK? I'm not saying everyone there is that way. My gut feeling, however, is that it is more slanted that way. It is also a bit heavier on the algo chasing/aggressive SEOers. Again, not exclusively so, but a bit heavier that way.

SEW, I think, is a bit heavier on site owners with content sites. They might employ affiliates, but they have their own direct sites that earn in various ways.

Now when Google makes an algo shift, I think it tends to hit the third-party sites, if you will, harder.

Most of this is gut feel. But it comes from years of watching threads explode on changes at WMW while I'd be at SEW getting little feedback from my readers about problems. The changes weren't hitting them.

Florida was a key exception. You could say SEW readers might not have been sending tons of feedback because they were going elsewhere for answers, weren't smart enough to see they were getting hit, whatever. But when Florida came along, I did hear from my own readers. That was a big sign to me that a major, major change across the board had hit Google.

But other changes, they go completely unnoticed by some because they have no impact. There are going to be plenty of people not seeing page drops.

It might also be a more patient audience here. We know BigDaddy is new. It's still going to shake things out. It could be folks here are just saying let's let it cruise for a bit then do more high level looks when things have stablized.

You've also got crossover folks. If you're at WMW, and there's a hot thread going on the topic already, probably doesn't make you think you need to get the same going here. But similarly, we might get threads on stuff going that WMW doesn't do.

Hope that helps explain why it might not seem an issue here, to the best I can figure out. But I'd say try starting a new thread on the topic and see if you can kick off some discussion again here.

[egain]

Quite an interesting read on the UK based website the Register


http://www.theregister.co.uk/2006/05...igdaddy_chaos/

[sootledir]

Not all webmasters are "seething" about BigDaddy. Some of us have done really well.

[glengara]

*It might also be a more patient audience here *

I'd say it's mostly an acquired rather than a natural patience ;-)

[graywolf]

I'm seeing quite a bit of crawling peculiarities. 301's seems to take a while to get sorted out, usually it's a few weeks, but I have some pages that were moved in February that still list under the old URI (I moved them from example.com/page.html to example.com/page/ )

I also have a test page I created on Google pages in February that they picked up a few days later that's not in the index anymore

h**p://thegraywolf.googlepages.com/home

clearly no one will miss that page but it's an example I can use to show pages that have dropped out for no apparent reason. I'm not entirely clear if it's just a crawling issue or a low quality/spam fighting effort.

[clint7]

Originally Posted by sootledir

Not all webmasters are "seething" about BigDaddy. Some of us have done really well.

Be careful. They aren't finished yet. The deletion of our pages started immediately with Big Daddy - around 2 months ago. Others are only just starting to see the problem now. Google have a big index. It will take a while for their bug to get around to deleting everyone's pages.

The forums here at SEW seem to be just about the only place (aside from the media) that has remained largely oblivious to the BD problems up until now. Now that The Register has reported on it, I wouldn't be surprised to see this leak out into the mainstream media. Not before time either.

[softplus]

If you want to see some interesting bot-behaviour, watch your raw access logs.

Google is fetching files by date (getting 304's), using HEAD instead of GET to check files, re-crawling the same files 100's of times within minutes, ignoring pages (for weeks now), trying to fetch long-dead pages, etc.

I think some of their proxies have some serious problems .

Imagine that some of the proxies have access-problems and still have the initial seeds (say from last year) in their storage. That could explain a lot of the pecularities, but you should also be able to see it via Adsense (something I haven't noticed, but they now use the same proxies; or are sites with Adsense generally doing better?)

[clint7]

Or...maybe Danny...if you're honest, you guys were just caught napping.

There must be 15 Google threads on this forum started by people looking for advice about their pages suddenly disapearing from Google's index. To a one, those threads were pursued along the usual lines: get more backlinks, original content, and so on. As a result, a lot of people jumped through a lot of hoops that they really needn't have jumped through.

If this turns out to be just another, "been there, got the T-Shirt" Google update , then you guys can feel some sense of pride in your laid back, relaxed, completely-ignore-it-don't-even-mention-it attitude.

If, on the other hand, this turns out to be the biggest screw-up in Google's history (more than likely IMHO), then the fact that this forum ignored the obvious signs for more than two months is nothing to be proud of.

[shor]

Perhaps all it is different strokes for different folks. Stirring the folks at SEW up certainly accomplishes very little.

As Danny mentioned, different forums attract different users. I'm an in-house SEM guy with a big corporate, and I browse the SEW forums for its 'high-brow' discussion and cre8site for its market-centric approach. TW is a mini /. for the search community, etc etc. Each site targets a different audience. WMW has always been known for its huge G-index/algo update threads while TW/DP/Cre8/SEW typically don't attract the same level of activity.

On a personal note I haven't noticed BigDaddy affecting the industries I'm involved in, nor has it affected my competitors, hence I haven't really felt a need to air concerns or observations.

The best advice during a G-issue always seems to stay the same; stay patient, don't make drastic changes, and let people who can affect action (eg. Mr Cutts & Google feedback) know what is going on as concisely and precisely as possible.

[shor]

Originally Posted by clint7

Now that The Register has reported on it, I wouldn't be surprised to see this leak out into the mainstream media. Not before time either.

I wouldn't say that Register piece will spur mainstream publications into action. Schmidt is justifying Google's massive capital investment in new infrastructure, while the Register is using Schmidt's quote as a sensationalist headline - "Full-up Google choking on web spam? Buddy, can you spare a server?"

The journalist has taken Eric Schmidt's quote a bit out of context:
From the IHT

Google continued to make substantial capital investments, mainly in computer servers, networking equipment and its data centers. It spent $345 million on such items in the first quarter, more than double the level of last year. Yahoo, its closest rival, spent $142 million on capital expenses in the first quarter.

Referring to the sheer volume of Web site information, video and e-mail that Google's servers hold, Schmidt said: "Those machines are full. We have a huge machine crisis."

Versus

But the new algorithms may not be solely to blame. Google's chief executive Eric Schmidt has hinted at another reason for the recent chaos. In Google's earnings conference call last month, Schmidt was frank about the extent of the problem.

"Those machines are full," he said. "We have a huge machine crisis."

[Chris_D]

What Are Moderators?
Moderators oversee specific forums. They generally have the ability to edit and delete posts, move threads, and perform other manipulations. Becoming a moderator for a specific forum is usually rewarded to users who are particularly helpful and knowledgeable in the subject of the forum they are moderating.

http://forums.searchenginewatch.com/...ator_expl ain

Where have the mods been? Have we been napping?

I can't speak for all moderators - but I've been pretty busy with clients. Running a professional SEO/SEM agancy takes a fair bit of time - and I see being a moderator here as a way of 'giving back' to a community - and particularly to Danny - whose writings and insights have helped me a lot over the past 5 years.

I also participate at several other forums - been a member at WMW since 2001 etc.

Having seen and participated in 'update' threads over the past 5 years - I'm a bit over them. Someone once said 'worry about the stuff you can influence or change - don't worry about the stuff you can't'.

So I tend to spend my time communicating 'update' feedback and observations directly to the people who can change or influence it - rather than posting theories and observations here at SEW.

The person who could have 'answered' the questions correctly isn't a mod - its another member - Googleguy - and he is notoriously quiet during an update cycle. Watching the cake cooking isn't always the most productive use of your time.....

Many of us here also participate in many other forums because - as Shor & Danny noted - each forum has its own 'specialist' focus. 600 post 'update' threads aren't SEW. And many mods here may have also posted about big daddy at other forums and blogs. Why duplicate?

[dannysullivan]

Or...maybe Danny...if you're honest, you guys were just caught napping.

Gosh, Clint, I don't recall you popping in and starting a big thread on the massive Big Daddy losses. I'm not depending on the moderators to ferret out everything going on in the search engine world, here on the forums. The moderators are primarily working to help the community as a whole interact and share information here. And that community as a whole -- here -- hasn't seemed to be that freaked out over Big Daddy, judging from the posts. Either they're napping or quite possibly, it's not an issue for them.

There must be 15 Google threads on this forum started by people looking for advice about their pages suddenly disapearing from Google's index.

Again, gosh, pick any day you want before Big Daddy, and you'd get plenty of posts about the same thing. People are always posting about falling out of Google.

If this turns out to be just another, "been there, got the T-Shirt" Google update , then you guys can feel some sense of pride in your laid back, relaxed, completely-ignore-it-don't-even-mention-it attitude.

Which supposes that were were somehow trying to ignore it. Who is trying to ignore it? I just looked at the Google Web Search area. I'm sorry -- there just aren't a lot of posts from people screaming out they are having a big problem.

If, on the other hand, this turns out to be the biggest screw-up in Google's history (more than likely IMHO), then the fact that this forum ignored the obvious signs for more than two months is nothing to be proud of.

So let's be clear. From what I've read over at WMW, this seems to be mostly an issue hitting pages in the supplemental index. I could be wrong -- there are a lot of posts, and it's the usual havoc trying to make sense of them. But it seems like people are concerned they are in the supplemental and then seeing nothing in the supplemental.

Hey, bigger issue. If you're in the supplemental, you already had a problem with Google. Pages in there already weren't likely to tap into much traffic. Those are the afterthought pages, ones Google doesn't consider good enough to be in the real index. So it's a freakout that you were in the supplemental and now you aren't even there? Yeah, I can understand that -- but the bigger issue was why were you in the supplemental in the first place.

[clint7]

So let's be clear. From what I've read over at WMW, this seems to be mostly an issue hitting pages in the supplemental index. I could be wrong -- there are a lot of posts, and it's the usual havoc trying to make sense of them. But it seems like people are concerned they are in the supplemental and then seeing nothing in the supplemental.

No. That's not right. The "supplemental issue" grabbed most of the "headlines" (so to speak) in the beginning, even though the bigger problem of 99% of many website's pages simply being deleted altogether was also there from the beginning. The supplemental issue got largely sorted (didn't experience that one myself) but the enormous numbers of deleted pages problem remains and seems to be spreading by the day.

Only those who have experienced the problem first hand are truly in a position to know how clearly bug-ridden this deletion/filtering process is. I can tell you that there is no easily discernable rhyme or reason to distinguish the pages that are in from those that are out.

Anyway. Just thought I should clear up that misunderstanding. Obviously if the problem only effected supplemental pages then this would be far less serious.

[dannysullivan]

Following up further, I've also just done a review of the Google forum. In all of April, we had some very brief discussions and reports of pages mysteriously disappearing in these threads:

Google where did we go? (comments please!)
site completely lost in Google
Pages dropping out of index
Why have 99% of my indexed Google pages
Indexed Pages Lost on Google: Happened to You?
Home Page put into Supplemental Results
Index Saturation Update

Problem with Google indexing secure pages, dropping whole site is also a longer discussion, though the person reporting the issue is aware of the Bigdaddy reports but not sure if that's his problem. And in fact, in some of the posts above, it wasn't always clear these were Bigdaddy issues.

Clint, several of these threads, you've made comments like:

Check some of the other SE/Webmaster forums. You are far from alone in this. The recent Big Daddy "infrastrucure" update is to blame. It's been steadilly wiping websites off the face of the earth (well, the Google index anyway) since around late February, early March.

Now you might very well be right. And if we've let you and others down by not getting a firm thread on the topic before now, my sincere apologies. All I can say is that the 10-15 threads you stated being out there on any day definitely are not there. I struggled to find the list above going back all through April.

It's simply not been an issue voiced here, perhaps because the discussion is already going well at WMW, perhaps because people aren't having the issue, perhaps because we didn't catch a trend somehow and get a unified thread -- I don't know. But we've got the thread going now, so folks having problems that want to talk, share, whatever here, please go for it. I'll even get it out on the home page as a featured thread.

FYI, there's a fresh, brief discussion here, Google Natural Traffic Down in April? , though I think this is now the thread to go forward in. Also, in Sandbox Release Preview? , someone found the supplemental issue has actually improved for them.

As for other resources Over in Google Groups, there's this thread: Google Sitemaps: Post here if you have lost index pages!.

WebmasterWorld has the key discussion going with lots of activity. It was posted above, but here it is again, Pages Dropping Out of Big Daddy Index.

There's also the Register article posted above, but here it is again: Full-up Google choking on web spam?.

As for that quote on the machine "crisis," I actually didn't find it taken out of context. If anything, that alarms me the most, that Google itself said there was a machine issue. I definitely missed seeing that nugget when it came out last month.

FYI, Google's been accused of being out of space before, but we've never had someone at Google itself talk about a crisis.

Google Out Of Index Space? for SEW paid members from Sept. 2004 talked about this. Here are some of the articles it mentioned:

Is Google Broken (Sept. 2004) & Is Google Broken (June 2003).

Now since you can see we've been told that Google's been broken years ago but somehow has managed to struggle on pretty successfully, you might understand why folks could be dubious about the latest claims.

But still -- to have Google's CEO himself saying there's a machine crisis, that's a big deal. The Bigdaddy drops could be coincidence, could be focused on particular types of sites or could indeed be a confirmation of problems.

Barry also did a summary of some additional forum threads on the blog yesterday in Google Results Suffering After "Big Daddy" Update?, That leads to his SE Roundtable post, Pages Coming In & Out of the Google.com Results & Google Index, which summarizes four threads at WMW and two at DigitalPoint.

Finally, those wanting to know more about Bigdaddy might check out New Google "Bigdaddy" Infrastructure Live, Data Center Open For Feedback

[sootledir]

The one site I manage that had the most dramatic effect went from 197,000 pages to 36. The reason I'm not worked up about it is, that site has only received 1% of its traffic problem due to whatever problem the site always.

It seems to me that the problem is finally exposed. The supplementals were deleted and it will take a few major crawl cycles to complete it.

I noticed this affected many VBulletins (they got rid of the many duplicate URLS automatically) and other websites. I've been lucky that it hasn't affected my income or traffic (both are up). It really seems to me there close to fixing some long-standing issues once and for all. The fact Cutts hasn't commented makes me think they're not done yet.

[sully]

Originally Posted by dannysullivan

My experience is different audiences; different activiities.

WMW, in my view, is heavily affiliate/adsensers/non-content site owners. IE, people who aren't running web sites with a lot of content but pushing people to sites with content and making money along the way.

DISCLAIMER!!!! The above is a gross overgeneralization, OK? I'm not saying everyone there is that way. My gut feeling, however, is that it is more slanted that way. It is also a bit heavier on the algo chasing/aggressive SEOers. Again, not exclusively so, but a bit heavier that way.

SEW, I think, is a bit heavier on site owners with content sites. They might employ affiliates, but they have their own direct sites that earn in various ways.

Now when Google makes an algo shift, I think it tends to hit the third-party sites, if you will, harder.

BigDaddy has hit many ecommerce sites. The products we (and thousands of others) sell are very similar, actually same type but style names and manufacturers are often the only differentiating factors. Therefore, most descriptions (product pages) are essentially the same. If we add "unique" content to those product pages, we would be doing it for search engines, not our users. And it would only annoy users to include content designed for search engines as there is nothing "unique" to say about it.

Shoppers just want the facts and they want to be able to find them quickly. Legitimate sites like this seem to have been placed in the same heap with affiliate and spam sites. We understand the search engines' dilemma with such sites, but there must be a better way to deal with it. We've seen some minor improvement over the past 2-3 days, but remain in the wait-and-see-mode (since February/March) to see how it all flushes out.