Have any of you used static replication of dynamic content for the purpose of serving spiders (and maybe users too) with static versions of your site rather than dynamic ones?

What technologies/solutions have you used? How well did it work and what kind of problems did you meet and solve?

I used to use the old Ultimate Bulletin Board forum softare (prior version 5.34) that produced static pages each time someone made a post. Great feature but the system was highly unstable (to say the least). This used to work very well for search engines (and still do, if you can stand the nightmare of keeping that software up and running :)). Since then, most software developers have moved to dynamic delivery of content rather than static page updates.

I actually liked it that way, with UBB. So low tech and nice. Good old static HTML pages always worked well in search engines (if well optimized) and I bet they continue to do so for many years. So, why do we not see more systems on the market that, at least, have this as an option for puplication of database content?

I'll give you a couple of examples from some of my sites :)

1st site www.agent-pro.com yeah so it's your same old ho hum directory... Nope. I've built it so that a realtor/seo/whomever wants to add a listing to the site can do it PLUS write an article about the city, area, restaurants, mortgage lenders that are prefered etc and it all get attached to their listing.

For an example follow the link arizona >> scottsdale from the index page and you'll see 2 realtors that have written articles about schools or real estate law. Real good info that is all dynamic all stored in a database but static. I got a mod rewrite kicked in to make the static pages work. I even wrote a little cms system to help the users write their articles.

If you wanna see how it works just sign up and add a url. doesn't even have to be your website as long as it's a real estate based site :)

<tangent>coolest thing about agent pro is the ArticleRank, yup you heard it not PR or WR but AR hehe, sites that have articles attached to them are given promminence over listings that don't have articles. A site with 4,000 links to it can still be crap BUT a site with good articles attached to it that gets regulated by a human = perfection! Point is this. Instead of a directory listing now you can have a listing that links to you plus a profile that links to you plus as many articles as you feel you can write that all LINK TO YOU!!! All dynamic all being done in the name of good quality content.</tangent>

site 2. www.mouse-house-tour.com

This site sits on a huge database of information. I've taken this and litterally made 2 directories 1 for the spiders one for the humans Both of which are visable and legit.

human = www.mouse-house-tour.com/results.php
spiders/human www.mouse-house-tour.com/listings_directory/

I could easily make them both static but since this isn't our bread and butter I can't devote too much time to it.

What it all boils down to is users come in fill out a fancy shmancy tour and it's added to the directory (spiderable area) automatically plus the links to the tours are all static too.

Same thing as site 1 powered by a mod rewrite.

Don't laugh, Mikkel, but I'm still running a forum loosely based on the old UBB 5.x forum software. With 1.8 million posts, it hasn't crashed once in something like four years. I doubt Ted would even recognize the code base as his any more, though. :)

Another site of mine carries the concept even further. Unlike the UBB, the database is static enough to not even reside on the server. Instead, the data sits in an Access DB on my local machine, and VBA spits out 18,000 pages of HTML code on demand.

Both sites are over five years old, designed when a top-of-the-line Linux box was a P400 with maybe 128M of RAM. Today's much more powerful servers need less hand-holding, which I think is why most developers have moved away from the static page paradigm.

IMO, the only real advantage to static pages is server load. Apache is *highly* optimized to output HTML and even a modest CPU can spit it out faster than most pipelines can handle. I honestly don't think the SEO issues are even relevant any more, since dynamic pages can so easily be designed to accommodate spiders today.

The single advantage of server load is offset by some pretty serious drawbacks. Caching remains a problem today, though much less so than five years ago. Change a static page and you never know which browsers and proxies will pick up your changes and which won't. The biggest disadvantage, however, is page regeneration.

Static pages created dynamically aren't really saving server load. It's just deferring it to a more convenient time. When I update the Access-driven site, it takes about two hours to upload all the files (it was running nearly ten hours when I was still on dial-up). I haven't regenerated all the pages on my forum in over two years and, frankly, would be afraid to even try considering the size of the database. I've essentially boxed myself into a if-it-ain't-broke-don't-fix-it corner from which there's no easy escape.

Dynamically created static pages will remain in my grab bag of tricks, but I seriously doubt I'll ever reach for it again. The problems they solved five or six years ago can now be better addressed with bigger, faster iron.

I promise I wont laugh ... (at least not out loud), Ron :)

One thing i loike about your desktop-generation is portability: You can easily move it to any host and server - Apache or NT, it dosen't matter, as your output is just HTML.

Yahoo! Stores such as ours (mexgrocer.com) which is dynamic on the backend and static on the front end. It works really great for us and the site never breaks . . . unless Y! has a hiccup.

Is it not easier just to cache the DB output?

The Smarty Template Engine (http://smarty.php.net) does this, as does Drupal (http://drupal.org).

It amounts to almost the same thing and is far less costly in time and effort ;-)

Nick

The smart thing about the off-line solution is that it is totally server independant - it can be uploaded to any server that can serve up HTML files.

Sure, just wget -r http://example.com and you're ready to whack that content up just about anywhere...

Nick