Search Engine Watch
SEO News

Go Back   Search Engine Watch Forums > Search Engines & Directories > Google > Google Web Search
FAQ Members List Calendar Forum Search Today's Posts Mark Forums Read

Reply
 
Thread Tools
Old 05-03-2006   #1
Relevancy
Relevancy Brings Results
 
Join Date: Jan 2005
Location: CA
Posts: 225
Relevancy is on a distinguished road
MODERATOR NOTE: I've merged posts from Where are all the SEW Moderators? relevant to the BigDaddy issues some are having into this thread.

No you are right Mods aren’t the only ones with answers that is why I had to go to webmasterworld to get into a in-depth discussion about this topic

http://www.webmasterworld.com/forum30/34055.htm

No one is really talking about it here. Few scattered threads here for such a huge issue.. I don't get it.

Last edited by dannysullivan : 05-05-2006 at 08:16 AM.
Relevancy is offline   Reply With Quote
Old 05-03-2006   #2
JohnW
 
JohnW's Avatar
 
Join Date: Jun 2004
Location: Virginia Beach, VA.
Posts: 976
JohnW has much to be proud ofJohnW has much to be proud ofJohnW has much to be proud ofJohnW has much to be proud ofJohnW has much to be proud ofJohnW has much to be proud ofJohnW has much to be proud ofJohnW has much to be proud ofJohnW has much to be proud ofJohnW has much to be proud of
There is a lot of discussion on some forums, but it doesn't go anywhere.

IMO what is being seen is just unplanned after-effects of BD. G rolled out some new technology to fix some things, and in the process they seem to have broken some other things. There isn't much to say about it other than it’s broken - it is what it is and will be over when it's over.

It does seem to be settling down, one DC at a time.
JohnW is offline   Reply With Quote
Old 05-03-2006   #3
Marcia
 
Marcia's Avatar
 
Join Date: Jun 2004
Location: Los Angeles, CA
Posts: 5,476
Marcia has a reputation beyond reputeMarcia has a reputation beyond reputeMarcia has a reputation beyond reputeMarcia has a reputation beyond reputeMarcia has a reputation beyond reputeMarcia has a reputation beyond reputeMarcia has a reputation beyond reputeMarcia has a reputation beyond reputeMarcia has a reputation beyond reputeMarcia has a reputation beyond reputeMarcia has a reputation beyond repute
Quote:
in-depth discussion about this topic
Have there been any in-depth comments and observations about how Google's *possible* use of historical data is related to some of the problems we're seeing right now? If so, please point me to the specific posts; I'd love to see what others may be spotting with this.

As for myself, I'm taking an in-depth look at several sites under 100 pages related to that very thing, as well as the handling of 301 and 302 redirects. I'd also like to be pointed to any specific posts related to this in those threads. I've read through and haven't seen any on either issue that might relate to what I'm seeing.

And Yahoo too, on penalties and redirect issues - good time to look at both engines and connect the dots with any issues that may cause related problems on both, or problems rooted in the same factors, even if they have different effects.
Marcia is offline   Reply With Quote
Old 05-03-2006   #4
JohnW
 
JohnW's Avatar
 
Join Date: Jun 2004
Location: Virginia Beach, VA.
Posts: 976
JohnW has much to be proud ofJohnW has much to be proud ofJohnW has much to be proud ofJohnW has much to be proud ofJohnW has much to be proud ofJohnW has much to be proud ofJohnW has much to be proud ofJohnW has much to be proud ofJohnW has much to be proud ofJohnW has much to be proud of
>I've read through and haven't seen any on either issue that might relate to what I'm seeing.

Exactly. But I have noticed things coming back to normal (or better) on certain DCs. It also looks like some sites are emerging for the first time on these same DCs.

Take a look at results on 64.233.167.104 and 72.14.207.104
JohnW is offline   Reply With Quote
Old 05-04-2006   #5
dannysullivan
Editor, SearchEngineLand.com (Info, Great Columns & Daily Recap Of Search News!)
 
Join Date: May 2004
Location: Search Engine Land
Posts: 2,085
dannysullivan has much to be proud ofdannysullivan has much to be proud ofdannysullivan has much to be proud ofdannysullivan has much to be proud ofdannysullivan has much to be proud ofdannysullivan has much to be proud ofdannysullivan has much to be proud ofdannysullivan has much to be proud of
Quote:
No one is really talking about it here. Few scattered threads here for such a huge issue.. I don't get it.
My experience is different audiences; different activiities.

WMW, in my view, is heavily affiliate/adsensers/non-content site owners. IE, people who aren't running web sites with a lot of content but pushing people to sites with content and making money along the way.

DISCLAIMER!!!! The above is a gross overgeneralization, OK? I'm not saying everyone there is that way. My gut feeling, however, is that it is more slanted that way. It is also a bit heavier on the algo chasing/aggressive SEOers. Again, not exclusively so, but a bit heavier that way.

SEW, I think, is a bit heavier on site owners with content sites. They might employ affiliates, but they have their own direct sites that earn in various ways.

Now when Google makes an algo shift, I think it tends to hit the third-party sites, if you will, harder.

Most of this is gut feel. But it comes from years of watching threads explode on changes at WMW while I'd be at SEW getting little feedback from my readers about problems. The changes weren't hitting them.

Florida was a key exception. You could say SEW readers might not have been sending tons of feedback because they were going elsewhere for answers, weren't smart enough to see they were getting hit, whatever. But when Florida came along, I did hear from my own readers. That was a big sign to me that a major, major change across the board had hit Google.

But other changes, they go completely unnoticed by some because they have no impact. There are going to be plenty of people not seeing page drops.

It might also be a more patient audience here. We know BigDaddy is new. It's still going to shake things out. It could be folks here are just saying let's let it cruise for a bit then do more high level looks when things have stablized.

You've also got crossover folks. If you're at WMW, and there's a hot thread going on the topic already, probably doesn't make you think you need to get the same going here. But similarly, we might get threads on stuff going that WMW doesn't do.

Hope that helps explain why it might not seem an issue here, to the best I can figure out. But I'd say try starting a new thread on the topic and see if you can kick off some discussion again here.
dannysullivan is offline   Reply With Quote
Old 05-04-2006   #6
egain
Member
 
Join Date: May 2005
Location: Lancashire, UK
Posts: 134
egain is on a distinguished road
Google Bigdaddy chaos - The Register

Quite an interesting read on the UK based website the Register


http://www.theregister.co.uk/2006/05...igdaddy_chaos/
egain is offline   Reply With Quote
Old 05-04-2006   #7
sootledir
directory.sootle.com
 
Join Date: Mar 2005
Posts: 146
sootledir has a spectacular aura aboutsootledir has a spectacular aura about
Thumbs up

Not all webmasters are "seething" about BigDaddy. Some of us have done really well.
sootledir is offline   Reply With Quote
Old 05-04-2006   #8
glengara
Member
 
Join Date: Nov 2004
Location: Done Leery
Posts: 1,118
glengara has much to be proud ofglengara has much to be proud ofglengara has much to be proud ofglengara has much to be proud ofglengara has much to be proud ofglengara has much to be proud ofglengara has much to be proud ofglengara has much to be proud of
*It might also be a more patient audience here *

I'd say it's mostly an acquired rather than a natural patience ;-)
glengara is offline   Reply With Quote
Old 05-04-2006   #9
graywolf
Member
 
Join Date: Jul 2004
Posts: 13
graywolf will become famous soon enough
I'm seeing quite a bit of crawling peculiarities. 301's seems to take a while to get sorted out, usually it's a few weeks, but I have some pages that were moved in February that still list under the old URI (I moved them from example.com/page.html to example.com/page/ )

I also have a test page I created on Google pages in February that they picked up a few days later that's not in the index anymore

h**p://thegraywolf.googlepages.com/home

clearly no one will miss that page but it's an example I can use to show pages that have dropped out for no apparent reason. I'm not entirely clear if it's just a crawling issue or a low quality/spam fighting effort.
graywolf is offline   Reply With Quote
Old 05-04-2006   #10
clint7
Member
 
Join Date: Apr 2006
Posts: 11
clint7 is on a distinguished road
Quote:
Originally Posted by sootledir
Not all webmasters are "seething" about BigDaddy. Some of us have done really well.
Be careful. They aren't finished yet. The deletion of our pages started immediately with Big Daddy - around 2 months ago. Others are only just starting to see the problem now. Google have a big index. It will take a while for their bug to get around to deleting everyone's pages.

The forums here at SEW seem to be just about the only place (aside from the media) that has remained largely oblivious to the BD problems up until now. Now that The Register has reported on it, I wouldn't be surprised to see this leak out into the mainstream media. Not before time either.
clint7 is offline   Reply With Quote
Old 05-04-2006   #11
softplus
old-n-grey
 
Join Date: May 2005
Location: CH(eeseland)
Posts: 48
softplus is on a distinguished road
If you want to see some interesting bot-behaviour, watch your raw access logs.

Google is fetching files by date (getting 304's), using HEAD instead of GET to check files, re-crawling the same files 100's of times within minutes, ignoring pages (for weeks now), trying to fetch long-dead pages, etc.

I think some of their proxies have some serious problems .

Imagine that some of the proxies have access-problems and still have the initial seeds (say from last year) in their storage. That could explain a lot of the pecularities, but you should also be able to see it via Adsense (something I haven't noticed, but they now use the same proxies; or are sites with Adsense generally doing better?)
softplus is offline   Reply With Quote
Old 05-04-2006   #12
clint7
Member
 
Join Date: Apr 2006
Posts: 11
clint7 is on a distinguished road
Or...maybe Danny...if you're honest, you guys were just caught napping.

There must be 15 Google threads on this forum started by people looking for advice about their pages suddenly disapearing from Google's index. To a one, those threads were pursued along the usual lines: get more backlinks, original content, and so on. As a result, a lot of people jumped through a lot of hoops that they really needn't have jumped through.

If this turns out to be just another, "been there, got the T-Shirt" Google update , then you guys can feel some sense of pride in your laid back, relaxed, completely-ignore-it-don't-even-mention-it attitude.

If, on the other hand, this turns out to be the biggest screw-up in Google's history (more than likely IMHO), then the fact that this forum ignored the obvious signs for more than two months is nothing to be proud of.
clint7 is offline   Reply With Quote
Old 05-05-2006   #13
shor
aka Lucas Ng. Aussie online marketer.
 
Join Date: Aug 2004
Posts: 161
shor is a jewel in the roughshor is a jewel in the roughshor is a jewel in the roughshor is a jewel in the rough
Perhaps all it is different strokes for different folks. Stirring the folks at SEW up certainly accomplishes very little.

As Danny mentioned, different forums attract different users. I'm an in-house SEM guy with a big corporate, and I browse the SEW forums for its 'high-brow' discussion and cre8site for its market-centric approach. TW is a mini /. for the search community, etc etc. Each site targets a different audience. WMW has always been known for its huge G-index/algo update threads while TW/DP/Cre8/SEW typically don't attract the same level of activity.

On a personal note I haven't noticed BigDaddy affecting the industries I'm involved in, nor has it affected my competitors, hence I haven't really felt a need to air concerns or observations.

The best advice during a G-issue always seems to stay the same; stay patient, don't make drastic changes, and let people who can affect action (eg. Mr Cutts & Google feedback) know what is going on as concisely and precisely as possible.
shor is offline   Reply With Quote
Old 05-05-2006   #14
shor
aka Lucas Ng. Aussie online marketer.
 
Join Date: Aug 2004
Posts: 161
shor is a jewel in the roughshor is a jewel in the roughshor is a jewel in the roughshor is a jewel in the rough
Quote:
Originally Posted by clint7
Now that The Register has reported on it, I wouldn't be surprised to see this leak out into the mainstream media. Not before time either.
I wouldn't say that Register piece will spur mainstream publications into action. Schmidt is justifying Google's massive capital investment in new infrastructure, while the Register is using Schmidt's quote as a sensationalist headline - "Full-up Google choking on web spam? Buddy, can you spare a server?"

The journalist has taken Eric Schmidt's quote a bit out of context:
From the IHT
Quote:
Google continued to make substantial capital investments, mainly in computer servers, networking equipment and its data centers. It spent $345 million on such items in the first quarter, more than double the level of last year. Yahoo, its closest rival, spent $142 million on capital expenses in the first quarter.

Referring to the sheer volume of Web site information, video and e-mail that Google's servers hold, Schmidt said: "Those machines are full. We have a huge machine crisis."
Versus
Quote:
But the new algorithms may not be solely to blame. Google's chief executive Eric Schmidt has hinted at another reason for the recent chaos. In Google's earnings conference call last month, Schmidt was frank about the extent of the problem.

"Those machines are full," he said. "We have a huge machine crisis."
shor is offline   Reply With Quote
Old 05-05-2006   #15
Chris_D
 
Chris_D's Avatar
 
Join Date: Jun 2004
Location: Sydney Australia
Posts: 1,099
Chris_D has much to be proud ofChris_D has much to be proud ofChris_D has much to be proud ofChris_D has much to be proud ofChris_D has much to be proud ofChris_D has much to be proud ofChris_D has much to be proud ofChris_D has much to be proud ofChris_D has much to be proud of
Quote:
What Are Moderators?
Moderators oversee specific forums. They generally have the ability to edit and delete posts, move threads, and perform other manipulations. Becoming a moderator for a specific forum is usually rewarded to users who are particularly helpful and knowledgeable in the subject of the forum they are moderating.
http://forums.searchenginewatch.com/...ator_expl ain

Where have the mods been? Have we been napping?

I can't speak for all moderators - but I've been pretty busy with clients. Running a professional SEO/SEM agancy takes a fair bit of time - and I see being a moderator here as a way of 'giving back' to a community - and particularly to Danny - whose writings and insights have helped me a lot over the past 5 years.

I also participate at several other forums - been a member at WMW since 2001 etc.

Having seen and participated in 'update' threads over the past 5 years - I'm a bit over them. Someone once said 'worry about the stuff you can influence or change - don't worry about the stuff you can't'.

So I tend to spend my time communicating 'update' feedback and observations directly to the people who can change or influence it - rather than posting theories and observations here at SEW.

The person who could have 'answered' the questions correctly isn't a mod - its another member - Googleguy - and he is notoriously quiet during an update cycle. Watching the cake cooking isn't always the most productive use of your time.....

Many of us here also participate in many other forums because - as Shor & Danny noted - each forum has its own 'specialist' focus. 600 post 'update' threads aren't SEW. And many mods here may have also posted about big daddy at other forums and blogs. Why duplicate?
Chris_D is offline   Reply With Quote
Old 05-05-2006   #16
dannysullivan
Editor, SearchEngineLand.com (Info, Great Columns & Daily Recap Of Search News!)
 
Join Date: May 2004
Location: Search Engine Land
Posts: 2,085
dannysullivan has much to be proud ofdannysullivan has much to be proud ofdannysullivan has much to be proud ofdannysullivan has much to be proud ofdannysullivan has much to be proud ofdannysullivan has much to be proud ofdannysullivan has much to be proud ofdannysullivan has much to be proud of
Quote:
Or...maybe Danny...if you're honest, you guys were just caught napping.
Gosh, Clint, I don't recall you popping in and starting a big thread on the massive Big Daddy losses. I'm not depending on the moderators to ferret out everything going on in the search engine world, here on the forums. The moderators are primarily working to help the community as a whole interact and share information here. And that community as a whole -- here -- hasn't seemed to be that freaked out over Big Daddy, judging from the posts. Either they're napping or quite possibly, it's not an issue for them.

Quote:
There must be 15 Google threads on this forum started by people looking for advice about their pages suddenly disapearing from Google's index.
Again, gosh, pick any day you want before Big Daddy, and you'd get plenty of posts about the same thing. People are always posting about falling out of Google.

Quote:
If this turns out to be just another, "been there, got the T-Shirt" Google update , then you guys can feel some sense of pride in your laid back, relaxed, completely-ignore-it-don't-even-mention-it attitude.
Which supposes that were were somehow trying to ignore it. Who is trying to ignore it? I just looked at the Google Web Search area. I'm sorry -- there just aren't a lot of posts from people screaming out they are having a big problem.

Quote:
If, on the other hand, this turns out to be the biggest screw-up in Google's history (more than likely IMHO), then the fact that this forum ignored the obvious signs for more than two months is nothing to be proud of.
So let's be clear. From what I've read over at WMW, this seems to be mostly an issue hitting pages in the supplemental index. I could be wrong -- there are a lot of posts, and it's the usual havoc trying to make sense of them. But it seems like people are concerned they are in the supplemental and then seeing nothing in the supplemental.

Hey, bigger issue. If you're in the supplemental, you already had a problem with Google. Pages in there already weren't likely to tap into much traffic. Those are the afterthought pages, ones Google doesn't consider good enough to be in the real index. So it's a freakout that you were in the supplemental and now you aren't even there? Yeah, I can understand that -- but the bigger issue was why were you in the supplemental in the first place.
dannysullivan is offline   Reply With Quote
Old 05-05-2006   #17
clint7
Member
 
Join Date: Apr 2006
Posts: 11
clint7 is on a distinguished road
Quote:
So let's be clear. From what I've read over at WMW, this seems to be mostly an issue hitting pages in the supplemental index. I could be wrong -- there are a lot of posts, and it's the usual havoc trying to make sense of them. But it seems like people are concerned they are in the supplemental and then seeing nothing in the supplemental.
No. That's not right. The "supplemental issue" grabbed most of the "headlines" (so to speak) in the beginning, even though the bigger problem of 99% of many website's pages simply being deleted altogether was also there from the beginning. The supplemental issue got largely sorted (didn't experience that one myself) but the enormous numbers of deleted pages problem remains and seems to be spreading by the day.

Only those who have experienced the problem first hand are truly in a position to know how clearly bug-ridden this deletion/filtering process is. I can tell you that there is no easily discernable rhyme or reason to distinguish the pages that are in from those that are out.

Anyway. Just thought I should clear up that misunderstanding. Obviously if the problem only effected supplemental pages then this would be far less serious.
clint7 is offline   Reply With Quote
Old 05-05-2006   #18
dannysullivan
Editor, SearchEngineLand.com (Info, Great Columns & Daily Recap Of Search News!)
 
Join Date: May 2004
Location: Search Engine Land
Posts: 2,085
dannysullivan has much to be proud ofdannysullivan has much to be proud ofdannysullivan has much to be proud ofdannysullivan has much to be proud ofdannysullivan has much to be proud ofdannysullivan has much to be proud ofdannysullivan has much to be proud ofdannysullivan has much to be proud of
Following up further, I've also just done a review of the Google forum. In all of April, we had some very brief discussions and reports of pages mysteriously disappearing in these threads:

Google where did we go? (comments please!)
site completely lost in Google
Pages dropping out of index
Why have 99% of my indexed Google pages
Indexed Pages Lost on Google: Happened to You?
Home Page put into Supplemental Results
Index Saturation Update

Problem with Google indexing secure pages, dropping whole site is also a longer discussion, though the person reporting the issue is aware of the Bigdaddy reports but not sure if that's his problem. And in fact, in some of the posts above, it wasn't always clear these were Bigdaddy issues.

Clint, several of these threads, you've made comments like:

Quote:
Check some of the other SE/Webmaster forums. You are far from alone in this. The recent Big Daddy "infrastrucure" update is to blame. It's been steadilly wiping websites off the face of the earth (well, the Google index anyway) since around late February, early March.
Now you might very well be right. And if we've let you and others down by not getting a firm thread on the topic before now, my sincere apologies. All I can say is that the 10-15 threads you stated being out there on any day definitely are not there. I struggled to find the list above going back all through April.

It's simply not been an issue voiced here, perhaps because the discussion is already going well at WMW, perhaps because people aren't having the issue, perhaps because we didn't catch a trend somehow and get a unified thread -- I don't know. But we've got the thread going now, so folks having problems that want to talk, share, whatever here, please go for it. I'll even get it out on the home page as a featured thread.

FYI, there's a fresh, brief discussion here, Google Natural Traffic Down in April? , though I think this is now the thread to go forward in. Also, in Sandbox Release Preview? , someone found the supplemental issue has actually improved for them.

As for other resources Over in Google Groups, there's this thread: Google Sitemaps: Post here if you have lost index pages!.

WebmasterWorld has the key discussion going with lots of activity. It was posted above, but here it is again, Pages Dropping Out of Big Daddy Index.

There's also the Register article posted above, but here it is again: Full-up Google choking on web spam?.

As for that quote on the machine "crisis," I actually didn't find it taken out of context. If anything, that alarms me the most, that Google itself said there was a machine issue. I definitely missed seeing that nugget when it came out last month.

FYI, Google's been accused of being out of space before, but we've never had someone at Google itself talk about a crisis.

Google Out Of Index Space? for SEW paid members from Sept. 2004 talked about this. Here are some of the articles it mentioned:

Is Google Broken (Sept. 2004) & Is Google Broken (June 2003).

Now since you can see we've been told that Google's been broken years ago but somehow has managed to struggle on pretty successfully, you might understand why folks could be dubious about the latest claims.

But still -- to have Google's CEO himself saying there's a machine crisis, that's a big deal. The Bigdaddy drops could be coincidence, could be focused on particular types of sites or could indeed be a confirmation of problems.

Barry also did a summary of some additional forum threads on the blog yesterday in Google Results Suffering After "Big Daddy" Update?, That leads to his SE Roundtable post, Pages Coming In & Out of the Google.com Results & Google Index, which summarizes four threads at WMW and two at DigitalPoint.

Finally, those wanting to know more about Bigdaddy might check out New Google "Bigdaddy" Infrastructure Live, Data Center Open For Feedback
dannysullivan is offline   Reply With Quote
Old 05-05-2006   #19
sootledir
directory.sootle.com
 
Join Date: Mar 2005
Posts: 146
sootledir has a spectacular aura aboutsootledir has a spectacular aura about
The one site I manage that had the most dramatic effect went from 197,000 pages to 36. The reason I'm not worked up about it is, that site has only received 1% of its traffic problem due to whatever problem the site always.

It seems to me that the problem is finally exposed. The supplementals were deleted and it will take a few major crawl cycles to complete it.

I noticed this affected many VBulletins (they got rid of the many duplicate URLS automatically) and other websites. I've been lucky that it hasn't affected my income or traffic (both are up). It really seems to me there close to fixing some long-standing issues once and for all. The fact Cutts hasn't commented makes me think they're not done yet.
sootledir is offline   Reply With Quote
Old 05-05-2006   #20
sully
Member
 
Join Date: Nov 2004
Posts: 50
sully is on a distinguished road
Quote:
Originally Posted by dannysullivan
My experience is different audiences; different activiities.

WMW, in my view, is heavily affiliate/adsensers/non-content site owners. IE, people who aren't running web sites with a lot of content but pushing people to sites with content and making money along the way.

DISCLAIMER!!!! The above is a gross overgeneralization, OK? I'm not saying everyone there is that way. My gut feeling, however, is that it is more slanted that way. It is also a bit heavier on the algo chasing/aggressive SEOers. Again, not exclusively so, but a bit heavier that way.

SEW, I think, is a bit heavier on site owners with content sites. They might employ affiliates, but they have their own direct sites that earn in various ways.

Now when Google makes an algo shift, I think it tends to hit the third-party sites, if you will, harder.
BigDaddy has hit many ecommerce sites. The products we (and thousands of others) sell are very similar, actually same type but style names and manufacturers are often the only differentiating factors. Therefore, most descriptions (product pages) are essentially the same. If we add "unique" content to those product pages, we would be doing it for search engines, not our users. And it would only annoy users to include content designed for search engines as there is nothing "unique" to say about it.

Shoppers just want the facts and they want to be able to find them quickly. Legitimate sites like this seem to have been placed in the same heap with affiliate and spam sites. We understand the search engines' dilemma with such sites, but there must be a better way to deal with it. We've seen some minor improvement over the past 2-3 days, but remain in the wait-and-see-mode (since February/March) to see how it all flushes out.
sully is offline   Reply With Quote
Reply


Currently Active Users Viewing This Thread: 1 (0 members and 1 guests)
 
Thread Tools

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

vB code is On
Smilies are On
[IMG] code is On
HTML code is Off