URL Rewrite - Custom 404

[Branden]

Greetings,

Need some advice. I am running ISAPI rewrite on a website (built with cold fusion mx , mysql). Recently the rewrite stopped working for whatever reason. I didn't catch it right away because it was the weekend.

What happened is that Google came by while the pages using the rewrite were down and I lost all my pages from its index becuase Google was just seeing my custom 404. This has been catastrophic - several thousands visitors and several thousands of dollars lost.

So..my question is - instead of sending visitors to a custom 404 page when I have a problem, is it better to throw a standard 404. Will Google then give me time until their next re-index to fix the problem?

Any help would be appreciated.

[rogerd]

I wouldn't count on Google to keep pages in its index after Googlebot hits 404s on those pages.

A couple of thoughts:
1) If this is a site with decent revenue, you might set up a timed checking process that would alert you if the rewrite engine failed or, for that matter, if they dynamic pages weren't showing up due to some other problem.

2) I'm guessing this would be more trouble than it's worth, but you could code your custom 404 page to be a backup for ISAPI Rewrite by parsing and translating the URL.

[Branden]

Thanks Roger,

It's too bad Google wouldn't give some sort of grace period before they dumped you. I am sure all sites have some sort of down time at one point. If they happen to come by when you are down it seems you are pretty much screwed until they visit again.

Which raises another question - Would it make sense that when Google recrawls the site I'll get back the majority of our previous positions?

Anyway.. I will definately look into some sort of a timed checking process that could email my mobile phone. There may be some services available like this on the market?

With the parsing - are you meaning to write the 404 to sniff to see if the redirect is not working - if not, deliver the actual query string driven page?

Great help Roger. Thanks for your time.

[rogerd]

Branden, I'm not saying that Google will instantly dump the pages when they come up 404, but I wouldn't want my income to rely on their cutting me some slack.

The 404 idea I mentioned is the old "404 trick" that lets you rewrite URLs on Windows servers without a server-installed program like ISAPI rewrite. In short, the custom 404 page includes code that parses the error URL and either (in the case of a truly bogus page) delivers an error message or, if the URL meets the criteria for a dynamic conversion, calls the dynamic page while returning appropriate server headers for the static URL. This isn't the preferred way to handle rewriting dynamic URLs, but if you lose your ISAPI rewrite from time to time it might be a backup. (You may have to modify the code of the dynamic pages, though, and that would entail additional messy work.)

[Branden]

thanks roger. appreciated.

[ThomasB]

Using Apache and PHP enables you to send a 200 header though it should return a 404.

Not sure if it works in IIS, too.

[rogerd]

Not sure if it works in IIS, too.

It works - that's the basis for the 404 trick. If everything is coded properly, the spider (or human visitor) sees an entirely normal static page despite all the behind-the-scenes action. Still, ISAPI rewrite or similar are preferred solutions. I haven't heard of many problems where the rewrite engine bombed out but the server kept running, and it's probably overkill to create a backup rewrite solution.

[Branden]

I am having the guys at Helicon and my hosting provider look into this. Hopefully we can figure out why the filter is dieing and the server keeps humming away. Only after reboot will it start to work again.

Anyway.. I'll let you all know what I find out in case you run into it again.