Can't get old site OUT of Google

[boxman]

I have a web site which I originally put up on a free web page provided by my ISP, Comcast, on a site like this:

http://home.comcast.net/~sitename

Several months ago I moved the site to a proper purchased domain on a commercial host.

My problem is that the old site is still indexed by Google -- even though Google's last cache of that site was in November 2004.

I've been trying to get the old site out of Google for months, but I can't seem to make that happen. Here's what I've done:

1. Following Google's instructions on http://www.google.com/webmasters/remove.html, I created a robots.txt file that disallows all robots.

2. Since after a few weeks the old site was still being indexed, I submitted the URL to Google using the form on http://www.google.com/addurl/?continue=/addurl. i was hoping that Google would try to crawl the site, find that it was excluded by robots.txt, then pull the site from its index. But that hasn't worked.

3. I've tried the "Automatic URL Removal Tool" at http://services.google.com:8882/urlconsole/controller . But when I follow the instructions, I get an error message that says:

"The following rule applies to a URL that is outside the jurisdiction of this robots.txt file: DISALLOW /"

I know that the usual recommendation when moving your site to a new URL is to place a 301 permanent redirect on the old site site. But Comcast doesn't allow users to set up 301 redirects. So I've put up a simple statement that the site has moved, and provided a link to the new one.

I'm at my wits end over this. Any suggestions?

[ruchit]

i am also having similar problem... I am working on my site and googlebot is hitting it 2-3 times per day (atleast this is what my web stats show me)... and the index is getting updated everyday in google's cache.. however rest of the site is still a month old.. and some pages even date back to November last year ... even though my site is well-linked internally.
Any clues?

[projectphp]

Dissallow: /

Oh, and just take down the old pages!

[ruchit]

..but my problem is little different.... i am updating those existing pages on daily basis these days.... but the cached copy is about 3 weeks old... the site is static in nature... googlebot visits me 2-3 times a day... but probably only hits homepage and moves out.

[boxman]

Originally Posted by projectphp

Dissallow: /

Oh, and just take down the old pages!

I already have the Disallow statement in robots.txt.

As for taking the old site down, I think that could make things worse. I've tried that on a temporary basis and found that the site returns a 403 Forbidden, according to http://gsitecrawler.com/tools/server-status.aspx, although users see some different text from Comcast about the site being unavailable. That's under Comcast's control, not mine. Also, for reasons I don't understand, the old site is ranked much higher than the new site in Google. I think that if I take the old site down, people looking for my site in Google will either get nothing (that is, they'll see Comcast's message that the site is unavailable), or, if they look at the cache, they will see what Google cached last November.

To try to make a long story short, I think there are three problems:

1. Google's recommend approach for removing a URL doesn't work on a Comcast personal home page. (Can anybody help me with that?)

2. Google seems to have stopped spidering the site, but hangs onto the November 2004 cache. And I can't seem to force it to visit the current version of the site. (Is Comcast blocking Google from visiting its customers' personal home pages? Or has Google decided not to spider those anymore?)

3. It's not possible to set up a 301 Permanent Redirect on a Comcast personal home page. (Or so I've read in Comcast's forums. Anybody know otherwise?)

[Chris_D]

Boxman wrote:

I already have the Disallow statement in robots.txt

Google's recommend approach for removing a URL doesn't work on a Comcast personal home page

To remove a URL, you firstly ask - then have the site return a 404 or block access (Robots.txt, meta robots etc).

The issue with a robots.txt is that it has to be located in the root folder for the domain - which you can't access.

Most of these types of sites really just limit your options to the "on page" solution = Meta robots tag.

<meta name="robots" content="noindex,follow">

Also - you change the links on the free isp to the depper pages of the new domain....