Getting the spider to go deeper

[lsredford]

For our bookstore, I've been able to (with help from you guys) get a pagerank of 5 and get 54,000 pages indexed by Google, but I"m still missing products. For example, if there are 150 thumbnails spread across 15 pages of Romance books (ten thumbnail book covers on each page) the spider is going though maybe the first four pages and stopping. Seems to be doing this in all of the categories. How can I make it go the the end?

My understanding is I can't expect more pages to be indexed unless my rank increases, so I should be choosy about what gets ranked right? Each book page has a link leading to a form where the customer can review the book. I"d rather another book be indexed rather than the review page, but the spider follows that link. How can I make it stop. Here's a sample page review page for a book if that helps.

http://www.diesel-ebooks.com/cgi-bin/writereview.cgi?item=parent-034542333X&pass=0

Thanks.

[lsredford]

easy now, don't everybody answer at once.

[seomike]

It's more like, where to start. If this was my store I'd bulldoze it to the ground and rebuild the entire link structure. I wouldn't let the mod rewrite go deeper than 1 or 2 directories. I'd add my book names into the urls and make them actual files with extensions instead of hanging directories. There are other little things I'd do but just a first glance it looks like a lot of work, it needs to be thoroughly planned so that the structure is consistant through the entire site. :D

[lsredford]

You'd bulldoze my baby to the ground? I know. I asked for feedback.... Upon further inspection, it was brought to my attention that the first page on my categories (which is indexed with no problem) has a different URL structure than the ones behind it so your point is well taken. I just need to figure out how to do this without having to rent a Caterpillar.

Thanks.

Scott

[ACSSEO]

You need to work on your URL strings. You need to remove all the extraneous characters in them so that the search engines can crawl deeper and with ease. You also want to think about getting more links to you internal pages from outside websites. This will help increase your pagerank on internal pages which will make Google crawl deeper. You also want to work on your sitemap. Look at About.com’s site map. (http://spiderbites.about.com/sitemap.htm) A better site map will help increase the number of indexed pages. I would not recommend copying About.com’s site map, but you can do something unique like them to help increase your indexed pages.

[lsredford]

hey, that site map is cool. Is there any kind of script out there that would go through a database and generate all of those?

Scott

[ACSSEO]

As far as I know there is not a ready made script that will do that. You probably will have to create your own script if you want it automatic.