Google indexes new blog pages, but not pages from main site

[mannersg]

Hello,

Perhaps someone can help me understand why Google is indexing (usually in a day or two) all new pages posted to my blog - but not (in over a month) pages that are added to the main "sister" site (which is created via Dreamweaver. Those are .htm pages that don't get pinged).

Also, the main page on this Dreamweaver-created "sister" site is indexed/updated by Google on a daily basis.Somehow, however, Google is not following the internal links on the main page - to those new pages I've been creating, usually just one directory level down from the main page. (I do have the robots.txt meta tag, with "index," "follow.")

I submitted a Google sitemap this past weekend. I'm quite sure the Googlebot has crawled the whole site since (as per the stats), but the new pages on the main site still haven't been indexed.

The site has more than 500 backlinks, page rank of 5 on the main page, it has been up for more than a year - and I just don't recall Google taking this long to index an .htm page before. The search rankings for the indexed pages are generally quite high, often among the top ten for the most important keywords on each page.

All the blog pages have links to the new pages on the main site, but that doesn't seem to be helping.

Anything I can do to speed things up for the indexing of the new pages on the main site?

Any assistance will be much appreciated.

MG

[vayapues]

Blog pages tend to get indexed very quickly. I am assuming you are pinging google everytime you update the blog.

The fact that you have been spidered does not mean that this data has been updated to all data centers.

Also, please clarify. You are using a robots.txt file, or a metatag?

You don't need a robots.txt file, but you definetly want to have it. It is critical that it be setup correctly, or you can do more harm than good. Perhaps you could post it, so others can take a look at it.

Also, you don't need to tell google to follow a link. You only need to tell them to not follow a link.

[mannersg]

Thank you so much for the reply.

I have a robots.txt metatag, not a file. (I actually thought they were one and the same. . .) And yes, the blog pages are automatically pinged via pingomatic.

Also, I'm happy to report that Google has indexed all those pages *today*. They've even indexed pages from the main "static" (as in, non-blog) site that I added just this past weekend and earlier this week.

Two things I did (in case this may help others):

a) Submitted my Google Sitemap. Sitemap was first submitted on Sat. night. Resubmitted yesterday. (I didn't know I had to resubmit it each time I made updates - I thought all I needed to do was to upload the changed sitemap to the server.)

Anyhow, from now on I'll be resubmitting the sitemap twice a day. I'm afraid they may think I'm a spammer if I do it more often. . .

b) I started manually pinging my "static" pages - the ones that don't belong to the blog. I'd never done that before. I'm not sure if that makes any difference, really, as I only began doing that yesterday, but who knows?

Thanks again for the response.

MG

[vayapues]

Anyhow, from now on I'll be resubmitting the sitemap twice a day. I'm afraid they may think I'm a spammer if I do it more often. . .

You have a few misconceptions. Please take this response for what it is worth.

Twice a day is way to often to submit anything to Google, or anyone else for that matter. Once is sufficient (Not once a day, but once ever). The only reason you might need to resubmit your sitemap, is if the sitemap itself actually changes. Google allows you to specify how frequently your pages change, within the sitemap itself. Google will then use that as a guideline for how often they should visit your site. It is not a guarantee they will visit as often as you tell them, just a guideline you provide to google. There is no benefit to your static pages to having them spidered frequently, if they are not updated frequently. It just wastes Google's resources, while giving you no benefit what-so-ever.

Pingomatic submits to various blog services, such as blogsearch, blogger, etc. It is intended as a service for actual blog entries, not static pages. Spamming these services will result in your getting a bad reputation, annoying a lot of people, and ultimately get you blacklisted.

Blog spam won't get you listed in the main google index any faster. You will get listed in blogsearch.google.com, but not in the main SERPs.

Your best bet is to write good solid original content, and work hard on getting solid inbound links from other sites in your industry. Even if you have some errors in other places, this will go along way.

[gehrlekrona]

When I create a new site i usually create a page with some information about the site and put up the page on one of my other sites so search engines can pick it up.
My problem is that it takes forever for google to index the new site, if ever. It seems like they right now thins that the page I created IS the new site. I have even had sites removed from the index but the page I created is still there. I am not sure what is going on with google but I am thinking NOT to allow google at all on my sites. Right now it seems that they have their algo backwards and allow spam sites and scraper sites but not the original sites in their index.... maybe someone shoulod theach their PhD's some real world experience? :eek:

[mannersg]

Once again, thanks for the responses.

I won't be pinging the static pages any longer. I didn't know that would be considered spamming - especially since a shorter version of the "static" pages do appear in my blog - with a link to the more thorough text found on the static page. But thanks for the warning. As I've just stated, I'll no longer do that.

Now, one thing I didn't quite understand about vayapues's Google comment. . .

I add new pages every day - and I make changes/additions to existing pages (it's a film site, heavy on data) - both on the main site and on the blog. Those changes are reflected on the sitemap via alterations/additions. When I add a new page, for instance, I add that page to the sitemap. At the end of the day, I ping Google to let it know that a new page is available for spidering.

I'd read somewhere - perhaps even in the Google Rules & Regulations page, but I'm really not sure - that one should ping Google whenever there's a change on the sitemap. Since I work on the site every day, that's what I've been doing once a day. (Instead of twice a day, as per my previous post.) Isn't that how it works?

As for "good solid content," that shouldn't be a problem as I'm a professional writer. (Though the "good" surely is relative.) The # of links to the site has been doubling about every couple of months or so - and the site has even been recommended by the "Los Angeles Times" (much to my surprise).

Thanks again for the response/advice.

MG

[vayapues]

Sounds like you are well on your way to a great site.

I add new pages every day - and I make changes/additions to existing pages (it's a film site, heavy on data) - both on the main site and on the blog. Those changes are reflected on the sitemap via alterations/additions. When I add a new page, for instance, I add that page to the sitemap. At the end of the day, I ping Google to let it know that a new page is available for spidering.
In this case you are absolutely correct, and justified. If their are new pages, and thus changes to the sitemap, than it is useful to update your sitemap. I misunderstood that you were sending them the same sitemap repeatedly.

[mannersg]

vayapues, thanks again for all your suggestions. And best of luck with your own online efforts!

MG