Mikkel deMib Svendsen
08-25-2004, 10:01 AM
I have worked on a couple of sites that have hundred of thousands or millions of good content pages and would like to share some experience in getting that many pages indexed, ranked well and not the least benchmarked.
In most cases I have been furtunate in the fact that these large sites have already had good or decent linkpopularity/PageRank. Prominent sites but with a site architecture so messed up that they hardly get any pages indexed and ranked.
Removing indexing barriers on such sites is often not a trival task. Not the least the "human element" can be a huge limitation in the radical changes it often takes. However, it is indeed possible. But be prepared: It takes time! And, I found that the kind of companies that operate million-page sites often require extensive documentation before they change as much as a comma in the web-coding.
Once the indexing barriers is removed I found it very important to implement a solid robots.txt file to block all the agent that you don't want in, and that respect this file. And that's actually quite a few. Hundreds. Imagine you have just "opened up" your website to spiders (removed the indexing barriers) and all these hundreds of agents starts to spider your millions of pages. Not good. Not good at all. This can in fact take down your servers or force you to invest in heavier servers and bandwidth upgrades. So get protected.
I have often used a sligthly modified version of Brett Tabkes robots.txt file that he publish under the GNU license. This one is very strict - I usually take a few of the bots off the list, but thats up to you.
www.webmasterworld.com/robots.txt
After the indexing barriers are removed the pages should get indexed, right? Well, they don't allways. At least not all of the them - the millions you have. At least, I found that it takes a lot of work on your linking structure and how you update the content (freshness). And this is a tricky part ...
It is not so difficult to create site maps on a site with a few hundred or thousand pages in a herarchy that makes sense and with a resonable number of levels. With million of pages that gets tricky. Often I find that some site-maps, and links to pages end up so deep that spiders don't prioritise them high enough in crawling.
There are a couple of things I found that helps: Theming areas and getting external links to the entry points of those areas. This way you can create a number of internal hubs that you link to from the main site map as well as create external links to. From those hubs you can start the nested site map for each section. This will give you fewer pages to deal with in each site-map hierachy.
Before this post gets too long, I want to hear others experience in getting that many pages indexed. Next we can go into how we rank them well too :)
In most cases I have been furtunate in the fact that these large sites have already had good or decent linkpopularity/PageRank. Prominent sites but with a site architecture so messed up that they hardly get any pages indexed and ranked.
Removing indexing barriers on such sites is often not a trival task. Not the least the "human element" can be a huge limitation in the radical changes it often takes. However, it is indeed possible. But be prepared: It takes time! And, I found that the kind of companies that operate million-page sites often require extensive documentation before they change as much as a comma in the web-coding.
Once the indexing barriers is removed I found it very important to implement a solid robots.txt file to block all the agent that you don't want in, and that respect this file. And that's actually quite a few. Hundreds. Imagine you have just "opened up" your website to spiders (removed the indexing barriers) and all these hundreds of agents starts to spider your millions of pages. Not good. Not good at all. This can in fact take down your servers or force you to invest in heavier servers and bandwidth upgrades. So get protected.
I have often used a sligthly modified version of Brett Tabkes robots.txt file that he publish under the GNU license. This one is very strict - I usually take a few of the bots off the list, but thats up to you.
www.webmasterworld.com/robots.txt
After the indexing barriers are removed the pages should get indexed, right? Well, they don't allways. At least not all of the them - the millions you have. At least, I found that it takes a lot of work on your linking structure and how you update the content (freshness). And this is a tricky part ...
It is not so difficult to create site maps on a site with a few hundred or thousand pages in a herarchy that makes sense and with a resonable number of levels. With million of pages that gets tricky. Often I find that some site-maps, and links to pages end up so deep that spiders don't prioritise them high enough in crawling.
There are a couple of things I found that helps: Theming areas and getting external links to the entry points of those areas. This way you can create a number of internal hubs that you link to from the main site map as well as create external links to. From those hubs you can start the nested site map for each section. This will give you fewer pages to deal with in each site-map hierachy.
Before this post gets too long, I want to hear others experience in getting that many pages indexed. Next we can go into how we rank them well too :)