404 Error Page/301 Redirect??

[critter]

Hello All..

This may sound like a newbie question.

In the past when I have taken over existing websites for SEO/SEM I always create a custom 404 page, or even re-direct that custom 404 page in my htaccess file to my homepage like so:

ErrorDocument 404 http://www.logicalsense.ca/

My question is should I use 301 redirect instead or is my custom 404 error page, or directing the 404 error page to my homepage ok in preventing Google and other engines from hitting old/non-existent pages?

Thanks

Critter

[Nick W]

That's what a 404 is 4 isnt it?

Nick

[critter]

Originally Posted by Nick W

That's what a 404 is 4 isnt it?

Nick

As I thought - but why does someone use a 301 Redirect instead of a 404?

I just want to be sure this is SEO friendly and won't pose any future problems..

Critter

[Nick W]

Well, a 301 is for permenantly moved right? So, you use a 301 to redirect to where you have moved the content of the old page.

If the page no longer exists, use 404. Just good webmastering, but good for bots and humans alike..

Nick

[I, Brian]

Actually, I'm getting confused on this issue, too.

I would normally set up a 404 as critter detailed, so when pages are removed, you can redirect to the homepage or sitemap with a 404 redirect in .htaccess.

However, recently I looked at using Google Sitemaps, and I'm told that this redirect is effectively sending 200 OK headers for pages removed from the site - so the removed pages are apparently still telling search engines that t
hey are there, while moving human users appropriately.

I tried setting up the 404 redirects as relative to the root domain - ie, "ErrorDocument 404 /" - but this shows the original URL on top of the page redirected to human users, and is effectively misleading them.

What is the best way to apply a .htaccess redirect so that you send proper 404 headers to human user and search engines, as well as show the new URL redirected to so that everyone's clear?

(Also, apologies for bringing up an old thread - I used Google site: search and this was the most appropriate of the first dozen listings).

[Chris_D]

Hi Brian,

It can be quite confusing.

Status code definitions http://www.w3.org/Protocols/rfc2616/rfc2616-sec10.html are worth reading if you are an incurable insomniac.

Basically, the server returns a status code in the http header in response to a page request - there are tons of different status codes (see link above).

This is over simplifying it - but basically - the server returns one status code per page request, and a page.

The 'top 4' responses are - the server either gives you the page you asked for - or- it sends you to the new page content location if it's moved -or- it can't find that page and gives you an 'error' page.

i.e.

200 - which indicates the server has found the page (url) you requested, and here it is;
301 - the page you requested has permanently moved to a new URL - so I'll give you the new URL content;
302 - the page you requested has temporarily moved to a new URL - so I'll give you the new URL content;
404 - The page you requested can't be found, I haven't got a clue where it could be, no one told me about that page, so sod off and guess again.

So:

set up a 404 as critter detailed, so when pages are removed, you can redirect to the homepage or sitemap with a 404 redirect in .htaccess.

The issue is that you can't correctly do both a 404 AND a redirect to another page - well - not without confusing spiders...

If a page is removed, or never existed (i.e. you removed it, they mistyped it) then serve up a custom 404 content page (which you configure in .htaccess). Have the custom 404 'content' page explain (on the custom 404 page) that the page is no longer there, its gone, and provide full nav or a mini site map (on the custom 404 error page.)

Its a semantic hair splitting, nit picking distinction - but:

- you should return a 404 header - with a special page of content to help the user.

Rather than:

- Returning a 200 ok - and then giving them a 404 "page" (without a 404 header)

or

- returning a 200 ok - and then doing a redirection (301, 302) to another page (like home or sitemap)

Search engines act on the header response. Give them a 404 header - they'll eventually stop requesting the page. Give them a 200 - and they keep coming back. Give them a 302 - and they'll index the original requested URL and the content of the redirection.

If you moved it - and it won't ever go back to the old URL (e.g. you converted the site from .asp to .php) - then use a 301 to take them to the correct new URL (again - in .htaccess) - so the new URL is indexed, and the old one is dropped.

So the .htaccess syntax for a customised 404 should be:

ErrorDocument 404 /brians-404-error-page.html

Where the url of your custom 404 page is brians-404-error-page.html and it is in the root directory.

Use http://www.rexswain.com/httpview.html to see the header you are returning.

Its pretty late here in Oz - does that help Brian? I've tried to explain it in broad terms - rather than cutting and pasting a complex RFC explanation.

[I, Brian]

That's a great help, Chris - thanks for that.

So much for working for human users, not for search engines.

[Chris_D]

No problem Brian. Sticky me if you get stuck with the .htaccess stuff.

So much for working for human users, not for search engines.

LOL!

[jamstooks]

First of all, Chris, thank you for that detailed explanation. I now do my 404's through .htaccess.

I have an interesting twist on this topic and was wondering if anyone new a way of doing this through htaccess, or any other way...

I will be moving a number of static pages over to dynamic pages and I'd like to 301 redirect them all. However, there are too many to redirect them individually. So, what I need is a page that checks the requested url to see if it matches the pattern of the old static pages and 301 redirects to the new url, or returns a 404 error if it doesn't match the pattern.

Also, a few of the old pages have dissappeared all together. How would Search Engines feel about a 301 redirect to a 404? If that's a problem I may need to access my database to check to url.

Is this even possible?

Thanks,
-Ben

[Chris_D]

Hi Ben

Sorry - missed your post.

I will be moving a number of static pages over to dynamic pages and I'd like to 301 redirect them all

Why not effectively keep the old static URLs - but make the content dynamic (i.e database driven)?

i.e. you can have a static URL; or a dynamically generated URL. You can have 'static' content; or you can have dynamically generated content.

Why not just keep the Static URLs - i.e. rewrite the new dynamic URLs into the old static ones? i.e. mod rewrite/ isapi rewrite (depending on your server flavour)

How would Search Engines feel about a 301 redirect to a 404?

A page request for an old URL (for a page that isn't redirected to the new page) should return a 404 by default.

[maildeepak]

Dear Chris..

thank you very much for your detailed explanation.

i also had these doubts and i started to verify my .htaccess after your detailed xplanation.

right now i have given a 404 redirect using errordocument code in my .htaccess.

ErrorDocument 404 http://www.domain.com/notfound.htm

but i checked a file which is not in the server(which should give a 404 error) using the tool u had specified and i got the following information.

Sending request:

GET /~arsenio/rockskis/?D=A HTTP/1.1
Host: www.domain.com
User-Agent: Mozilla/5.0 (Windows; U; Windows NT 5.0; en-US; rv:1.8.0.1) Gecko/20060111 Firefox/1.5.0.1
Connection: close

Header (Length = 364):

HTTP/1.1·302·Found[size=-1][color=#800080](CR)[/color][/size][size=-1][color=#0000ff](LF)[/color][/size]
Date:·Mon,·10·Apr·2006·10:17:16·GMT[size=-1][color=#800080](CR)[/color][/size][size=-1][color=#0000ff](LF)[/color][/size]
Server:[color=#0000ff][size=-1]I removed this part for security purposes.[/size][/color]
Location:·http://www.domain.com/notfound.htm[size=-1][color=#800080](CR)[/color][/size][size=-1][color=#0000ff](LF)[/color][/size]
Connection:·close[size=-1][color=#800080](CR)[/color][/size][size=-1][color=#0000ff](LF)[/color][/size]
Transfer-Encoding:·chunked[size=-1][color=#800080](CR)[/color][/size][size=-1][color=#0000ff](LF)[/color][/size]
Content-Type:·text/html;·charset=iso-8859-1[size=-1][color=#800080](CR)[/color][/size][size=-1][color=#0000ff](LF)[/color][/size]
[size=-1][color=#800080](CR)[/color][/size][size=-1][color=#0000ff](LF)[/color][/size]

as u can see from the blocked, italicsed and underlined part, it shows a 302 redirect. now i am confused. I now realize why i am getting large no. of 302 hits in my log file. it is bcos of this 404 redirection.

can u pls clarify on what has gone wrong? or is this normal?

regards,
deepak.c

[Chris_D]

Hi Deepak

Here are a few quick checks:

1. Is there anything else in your .htaccess file over riding the ErrorDocument 404?

2. Did you check the response using the Rex Swain tool? A few other 'similar' tools I've used don't always return the correct codes.

3. Does the .htaccess file have permissions set correctly? (telnet into your server and type chmod 644 .htaccess at the prompt).

[jamstooks]

Thanks Chris,

The old URL's are from an old ASP version of the site and many more topics have been added under the PHP design.

Here's an example of what I did for anyone who's interested:

Old pages were of the form "www.domain.com/topic_string.asp".

New site was of the form "www.domain.com/index.php?pt=topic&ts=topic_string".

I combined the two using .htacces and have both 301 redirecting to SEF versions of the PHP pages. The link is now "www.domain.com/t_topic_string".

Here's the .htaccess code:

Options +FollowSymlinks
RewriteEngine On

#########################
# 301 Redirects (Permanant Moves)
#########################

# Old ASP Pages
RedirectMatch 301 /(.*)\.asp$ http://www.domain.com/topic_$1

# Old PHP Pages
RewriteCond %{REQUEST_URI} ^/index.php$ 
RewriteCond %{QUERY_STRING} pt=topic&ts=(.*)
RewriteRule (.*) /cr_%1? [R=301,L]

#########################
# 404 Pages
#########################

ErrorDocument 404 /pnf.php

#########################
# MOD_REWRITES
#########################

# Topics
RewriteCond %{REQUEST_URI} ^/topic_(.*)	[NC] 
RewriteRule (.*) /newindex.php?pt=topic&ts=%1 [L]

The only shortcoming here is that I can't query the database for a removed topic. However, if newindex.php detects that the topic doesn't exist then it will return a 404 header via php. That makes it a 301 to a 404. I assume search engines can handle this.

I do wonder though, if search engines will hurt me because I have two different pages 301 redirecting to one page. Anyone know how this works and what rankings are kept?

Cheers,
-Ben

[maildeepak]

Originally Posted by Chris_D

Hi Deepak

Here are a few quick checks:

1. Is there anything else in your .htaccess file over riding the ErrorDocument 404?

RewriteEngine On
RewriteCond %{HTTP_HOST} ^domain.com
RewriteRule ^(.*)$ http://www.domain.com/$1 [r=301,L]

I am using the above set of code to redirect non-www to www version of the site. But i belive this is also giving me a 302 code only..

2. Did you check the response using the Rex Swain tool? A few other 'similar' tools I've used don't always return the correct codes.

yes, i checked the response using the Rex Swain tool only. The one you had referenced above.

3. Does the .htaccess file have permissions set correctly? (telnet into your server and type chmod 644 .htaccess at the prompt).

My system admin says that the permissions have been set to 644 only.

thanx for taking the time and helping us out..

my htaccess file looks like this:

RewriteEngine On
RewriteCond %{HTTP_HOST} ^domain.com
RewriteRule ^(.*)$ http://www.domain.com/$1 [r=301,L]

#error line
ErrorDocument 404 http://www.domain.com/notfound.htm

cheers,
deepak

[jamstooks]

Thanks for all the good info guys.

I created an extension for firefox called HeaderGetter that makes it easy to test the Headers for the current page or any link.

https://addons.mozilla.org/firefox/2526/

Thought it might be useful...

-Ben

[maildeepak]

Dear Chris,...

I am eagerly waiting for your help regarding this below post.

deepak

Originally Posted by maildeepak

RewriteEngine On
RewriteCond %{HTTP_HOST} ^domain.com
RewriteRule ^(.*)$ http://www.domain.com/$1 [r=301,L]

I am using the above set of code to redirect non-www to www version of the site. But i belive this is also giving me a 302 code only..



yes, i checked the response using the Rex Swain tool only. The one you had referenced above.


My system admin says that the permissions have been set to 644 only.

thanx for taking the time and helping us out..

my htaccess file looks like this:


cheers,
deepak

[jamstooks]

Deepak,

You need to use '$' in the first line to mark the end of the line:

RewriteCond %{HTTP_HOST} ^domain.com$
RewriteRule ^(.*)$ http://www.domain.com/$1 [r=301,L]

not

RewriteCond %{HTTP_HOST} ^domain.com
RewriteRule ^(.*)$ http://www.domain.com/$1 [r=301,L]

Good Luck,
-Ben

[AussieWebmaster]

there is a great article about 404 redirects and site migration