Matching SEO to Infrastructure Capacity

[unixbob]

I've just come back from the Search Engine Strategies conference in New York and found it very useful. I now have lots of ideas for how to improve the visibility of my already popular site. What I noticed at the conference was that most delegates were either marketeers or web developers. I work for the operations department and have a slightly different perspective on the effects of driving more traffic to our web servers. Marketing graphs which show 400% increases in traffic are great on slideshows but this can translate to a website which was quite happily serving 100,000 customers adequately but unable to serve 400,000 at all. And the cost to provide improved infrastructure to enable the website to deliver to all of the new potential customers may be proibitive for small firms.

Using my own site as an example, I've identified a few small changes that I think will have a large impact on the visibility of a key section of our site. However I'm also aware that there are performance issues with the application that drives that part of the site and so I'm holding off altering the keywords and improving the relevance of the page design until the application is capable of taking the additional traffic.

This brings me to my point. What I was unable to glean from the conference was a methodology for gauging the effectiveness of optimisations. I'd like to make small improvements to be able to show the value of SEO within the organisation, but I don't want to swamp the system so that it becomes unusable by 100,000 users instead of usable for 20,000 users.

I'm new to SEO, so if this is covered elsewhere or is a basic problem then please excuse my ignorance.

[JPnyc]

I was there too, it was quite eductational for me. I don't think that's all that common a problem, although I can surely be mistaken. I think if you're already getting traffic which is near the limit of your service ability, then you don't worry about SEO. SEO is a concern for people who aren't getting what they need and can handle, in terms of traffic. While I suppose it's possible to get more than you bargained for and find yourself swamped, I would bet it's less than an everyday occurrence. The net is a big place with lots of competition so I think just increasing traffic up to capacity is enough to hope for.

[unixbob]

I think that perhaps I'm expecting SEO to be a more exact science than it is - e.g. keyword rich titles are more effective than sitemaps. I'd like to be able to have a level of control over the effectiveness of the optimisations and be able to ensure that I don't induce a reduced quality of service by receiving unprecedented levels of traffic.

I know it sounds a little silly to want to do optimisations but not make them very effective. Our site can serve 6 million dynamic pages on a busy day and I'm just conscious that small changes can have dramatic effects. Perhaps I'm expecting the results of SEO to be more effective than they are realistically likely to be.

[JPnyc]

Well you can expect to see increases in traffic but in general, yes I would expect that a sudden flash flood that you'd be unable to handle isn't at all common or likely. And if numbers were ever to grow too large at some point, I would think it would be a long time coming, and a very gradual occurrence, rather than an explosion.

[risk]

Originally Posted by unixbob

I think that perhaps I'm expecting SEO to be a more exact science than it is - e.g. keyword rich titles are more effective than sitemaps. I'd like to be able to have a level of control over the effectiveness of the optimisations and be able to ensure that I don't induce a reduced quality of service by receiving unprecedented levels of traffic.

I know it sounds a little silly to want to do optimisations but not make them very effective. Our site can serve 6 million dynamic pages on a busy day and I'm just conscious that small changes can have dramatic effects. Perhaps I'm expecting the results of SEO to be more effective than they are realistically likely to be.

since i'm in operations as well, i suggest applying the usual engineering solutions to this problem. estimate a likely increase in peak traffic, build extra capacity to handle that and put in throttling mechanisms to maintain service levels to the first x concurrent users you can handle.

of course, building extra capacity can be non-trivial, eg if you need to go from a single mysql backend to an oracle rac solution ;] but hey, operations isn't easy - that's why we get the big bucks and all the chicks <grin>

-rsk

[projectphp]

Ok, thinking laterally, why not try to optimise the site's code to accomodate a possible boost in traffic? Things like utilising 304 response codes can save a great deal of badwidth and, even better, processing time.

Often, even in server intensive dynamic sites, not a heck of a lot changes on a page except for peripheral, non-ranking stuff, like banner ads, specials etc. If you ignore that, and only return pages when the actual content changes, you can save significant time / space.

Similarly, reducing code, removing CSS to seperate files, and using subdomains to host individual sections seperately can help, as can turning off real time monitoring features (like tracking every user) and moving these functions to third party providers, most of whom have better functionality anyway.

Very few sites I have ever seen run particularly efficiently, and minor changes to code can often solve larger problems caused by increased traffic.

I think that perhaps I'm expecting SEO to be a more exact science than it is..

I don't get that. Marketing an exact science, and niether is economics. In SEO in particular, the playing field is constantly changing. Not only do SEs change algos, but competitors change their sites frequently as well. Todays SEO hopeless site is tomorrows SEO giant. So really, any planning that depends upon organic SEo traffic is likely to be heavily flawed. So over estimate is my best advice