MyTeeSpot
05-15-2006, 11:43 AM
I just recently started doing google sitemaps. In the Diagnostic section under the HTTP errors there are some 404 errors from May 7, 10, 11 and 12th (I have since fixed these problems). Can I assume this means that I am being crawled by google semi-regularly?
And - why if i'm getting crawled so often are no pages except my homepage indexed (which has been for the 2 months my site has been live)?
Also - since my homepage (Dynamic (http://www.myteespot.com) ) goes straight to a dynamic page, i'm using my robots.txt file to only allow google to crawl my static site (Static (http://www.myteespot.com/catalog/index.html) ). I am since getting "We can't currently access your home page because of a robots.txt restriction." under the 'Potential Indexing Problems' in the Sitemaps Diagnostics summary. I don't care if my dynamic homepage is indexed - but will this problem affect indexing/pagerank of my static pages?
Any answers/feedback/hints/suggestions would be greatly apreciated.
Thanks.
And - why if i'm getting crawled so often are no pages except my homepage indexed (which has been for the 2 months my site has been live)?
Also - since my homepage (Dynamic (http://www.myteespot.com) ) goes straight to a dynamic page, i'm using my robots.txt file to only allow google to crawl my static site (Static (http://www.myteespot.com/catalog/index.html) ). I am since getting "We can't currently access your home page because of a robots.txt restriction." under the 'Potential Indexing Problems' in the Sitemaps Diagnostics summary. I don't care if my dynamic homepage is indexed - but will this problem affect indexing/pagerank of my static pages?
Any answers/feedback/hints/suggestions would be greatly apreciated.
Thanks.