301 Redirect stopping spider?

[KevinSource]

If I use a 301 redirect on the landing page of a domain(1) to point it to a different domain(2), but there are content pages on domain(1), will the spider stop searching that domain once the redirect is hit? In other words, am I eliminating the content on domain(1) from being indexed because the spider immediatley leaves this domain?

[Incubator]

YES. If you have good quality content on Domain 1, why are you re-directing to Domain 2 ? Why not market both domains with quality content that is indexable as opposed to using any re-directs at all? As long as they are not mirror sites you will be able to broaden the reach of the 2 sites with
themed, related content and updates across the 2 sites.

Cheers

Wc

[I, Brian]

So far as I understand the situation given, spiders won't even be given the content of domain 1 for indexing - the server will effectievly kick them onto domain 2 with a 301.

[Ron Carnell]

Search engines spider pages, not sites.

As long as there is a valid link to a page, the robot will follow the link and spider the page. Disallowing a page in your robots.txt doesn't stop the spider from visiting other pages, and it's exactly the same with redirects. Your 301 will prevent *that* page from being indexed, but it won't directly affect any other page on your web site. (It might "indirectly" affect them if the redirected page contains the only links to other pages.)

[KevinSource]

Thank you all for your input. I realized what you said Ron, that the spider will still follow the links and index the pages through this means. In chekcing the Googlebot crawl of my sites, I see that the redirect did not affect anything with indexing. Thanks.

Hello! I would like to make sure that we are not doing anything that might be considered spam in terms of our domain forwarding and redirecting. Can anyone point me towards an article so I can start learning the basics? I think the problem is that my server administrator uses different terminology that I am seeing here in the forums, so I can't be sure if we are complying with "best practices."

Thanks!

[Robert_Charlton]

Originally Posted by KevinSource

If I use a 301 redirect on the landing page of a domain(1) to point it to a different domain(2)....

Kevin - If your redirect is on the page (ie, in the page code), it's not a 301. Can't tell from your post, but if the redirect is actually on the page, you have a "client side" redirect... in all probability a meta refresh or javascript redirect executed in the browser.

301s are server side redirects, generally set up in the .htaccess file (if you have an Apache server), and using either the PermanentRedirect or mod_rewrite to redirect the page.

To respond also to Shenoa's concern, 301 server side redirects are the proper way to redirect domains or pages (and you can set up a 301 to redirect a whole domain, not just one page). I'm not a server expert and can't suggest any good introductory articles, but maybe someone else can.

Meta refresh redirects, or javascript redirects, set up on each page, are frowned upon by the search engines and can lead to lots of problems. I feel they should be avoided.

301s used to work with no problems in Google (and on the other engines), and generally after a few weeks Google would not only forward users but would redirect spiders and "transmit" PageRank and link relevancy. Apparently, since June or so, Google has been slow to reinstate PageRank on (some?/all?) sites using 301s to redirect to a new domain, resulting in a "sandbox" or "time lag" effect that's been discussed elsewhere. Old 301s have continued to work fine.

On the new Yahoo, they've been having problems with new 301s. A Yahoo engineer at SES told me he hopes that problem will be cleared up in a month or so.

I'd say that this is not a good time to change domain names and redirect a site. I'm guessing that things might get better on Google after the IPO.

If it is a 301 redirecting just that one page, Ron is correct that the other pages can be spidered if there are other links to them. The spider will not follow anything from the redirected page, though, and if that is a high PR page, you're really shooting yourself in the foot with this arrangement.

A PS for Shenoa... note that "redirecting" a site is not the same as moving a site (say to another IP), assuming the site structure remains the same, if that's what you might be thinking of doing. You may also be talking about pointing multiple domains at a site, eg. The vocabulary gets very fuzzy if you're not precise about what you're doing. In general, you do not want to have the same content appear under two different urls in the address bar. If that's what you mean by forwarding or redirecting domains, you're not doing it right... but that's way off topic for this thread. Registrar "pointing" or forwarding parked domains is yet another variant... most of them use 302s, and this generally won't lead to problems.