PDA

View Full Version : Crawled but not indexed?


MyTeeSpot
05-15-2006, 11:43 AM
I just recently started doing google sitemaps. In the Diagnostic section under the HTTP errors there are some 404 errors from May 7, 10, 11 and 12th (I have since fixed these problems). Can I assume this means that I am being crawled by google semi-regularly?

And - why if i'm getting crawled so often are no pages except my homepage indexed (which has been for the 2 months my site has been live)?

Also - since my homepage (Dynamic (http://www.myteespot.com) ) goes straight to a dynamic page, i'm using my robots.txt file to only allow google to crawl my static site (Static (http://www.myteespot.com/catalog/index.html) ). I am since getting "We can't currently access your home page because of a robots.txt restriction." under the 'Potential Indexing Problems' in the Sitemaps Diagnostics summary. I don't care if my dynamic homepage is indexed - but will this problem affect indexing/pagerank of my static pages?

Any answers/feedback/hints/suggestions would be greatly apreciated.

Thanks.

PhilC
05-15-2006, 12:32 PM
If the Sitemap has passed muster, it should be fetched on a regular basis. It doesn't mean that the pages will be crawled, but they probably will be sooner or later. It certainly doesn't mean that they are being crawled already. The way to know if they are being crawled is to check your log file, and even that won't tell you everything if you have AdSense in the site, because the normal crawler uses the AdSense crawler's cache of pages when it can, so pages can be indexed that have never been fetched by the normal googlebot.