New Verification For Google Sitemaps & Other Changes

[Relevancy]

Not sure how many have allready seen this, but G sitemaps now offers a verification feature that shows you what urls had problems being indexed.

You ave to upload a blank html file to your server with a Google id in the file name.

Any thoughts?

I tried it and it works good.

[Rob]

We noticed today that the main page of the Google sitemaps has changed - there are now "verify" links beside each sitemap url. When you click on the links you get:

Verify Site: http://www.site.com/

Google has crawled your site http://www.site.com/ and would like to show you some results. However, in order to make sure we don't expose private information about your site to others, we need to verify your ownership.

1. Create a verification file

Create an empty file named GOOGLEb1111add111fa8e7.html . This file uniquely identifies you for Google. You can create this file in any text editor. The file should be empty, since we are only checking that it exists at the same location as your Sitemap and we aren't going to read the contents. You can read more about this file here.
2. Upload the verification file

Once you have created the verification file, place it on your server at http://www.site.com/.

So I did this and uploaded the file. And got this error:

NOT VERIFIED
We've detected that your 404 (file not found) error page returns a status of 200 (OK) in the header.

When I click on this link I see:

We've detected that your 404 (file not found) error page returns a status of 200 (OK) in the header.
this configuration presents a security risk for site verification and therefore, we can't verify your site. If your web server is configured to return a status of 200 in the header of 404 pages, and we enabled you to verify your site with this configuration, others would be able to take advantage of this and verify your site as well. This would allow others to see your site statistics. To ensure that no one can take advantage of this configuration to view statistics to sites they don't own, we only verify sites that return a status of 404 in the header of 404 pages.

Please modify your web server configuration to return a status of 404 in the header of 404 pages. Once your web server is configured correctly, try to verify the site again. If your web server is configured this way and you receive this error, click Check Status again and we'll recheck your configuration.

I may go ahead and make the modifications to the server headers but I was wondering if anyone else has seen this and understands why Google is doing this now?

[softplus]

Error pages need to return an error return code - otherwise search engines will need to guess which pages are error and which are real... Just imagine if your error page just redirected to your entry-page: duplicate content. Now imagine if Google throws your entry page from the serps and instead keeps the error page (with a broken URL) - ouch!

IMHO this is way overdue. You can still make a nice error page and return 404, you can even return your entry page instead of an error page and return 404. The user will never know, but the search engine will.

Yahoo does this automatically when crawling, it will try a bogus URL and see what happens.

Have you seen the other changes? It even lists URLs which are linked externally and return errors -- nice!

[dannysullivan]

FYI, I merged two threads on this topic together.

This is a rather fantastic change from what I see. Rather then trying to "guess" what problems the spiders may have with your site, you can actually check. For example, I see that on one site we have a link /folder/) with a parentheses in the tag, another page has an extra / at the end, both of which cause errors for Google. This allows us, as webmasters, to identify and fix the problems that our sites have thus improving the spiders ability to crawl and hopefully resulting in more indexed pages with better rankings.

[smallbiz]

Originally Posted by softplus

Error pages need to return an error return code - otherwise search engines will need to guess which pages are error and which are real... Just imagine if your error page just redirected to your entry-page: duplicate content. Now imagine if Google throws your entry page from the serps and instead keeps the error page (with a broken URL) - ouch!

IMHO this is way overdue. You can still make a nice error page and return 404, you can even return your entry page instead of an error page and return 404. The user will never know, but the search engine will.

Yahoo does this automatically when crawling, it will try a bogus URL and see what happens.

Have you seen the other changes? It even lists URLs which are linked externally and return errors -- nice!

Will it identify pages that it considers duplicates? Thanks,