Google Sitemaps Now Accepting Web Page Feeds

[Marketing Guy]

Great new addition, particularly for new sites!

Any plans to add some guides to creating sitemaps for common dynamic content packages, such as forum, blog, etc software (like dyn4mik3 did for Wrodpress)? Given that Googlebot has some issues with sessions IDs for example, a PhPbb site map surely would be a huge boost to forum owners. I'm sure the respective for communities for commonly available software will be quick in knocking something together but I would be nice if G could as well!

MG

[Jorge]

It sounds great, but I don't see how it is not going to be a major change for some, good for some, bad for others. I myself have a medium sized site with a couple hundred pages in the G index, but if I can get the other 1500 I should see some changes in SERPs.

[critter]

Does the sitemap have to be .XLS (XML) File?

Cheers

Critter

[dannysullivan]

I want a non-python tool that people can use to generate stuff. I read about the tool, thought "cool," then went to download and fire it up. I quickly messaged the developers they they needed to look at it instead.

How about something on Google itself that would let me plug in my domain, then get back a list of all the URLs you are already have spidered and what the current frequency is like, that I could download say into Excel. I'd love to see things like:

+ URL
+ Page Title
+ Google's current frequency of crawl
+ Google's impression of how often it gets modified

Then I could sort and say, "Woah, that page changes a lot more than that," then use the recommended revisit column to push the recommended frequency of revisits upward.

That's the challenge right now. We've got thousands of pages here, and knowing where to begin in setting priorities is a big challenge. My gut feeling was to do nothing and trust Google is probably guessing right. But that type of info above would make it easier to know if it is not.

The tool could also show pages Google knows about but hasn't index. Then we could say, yes, please get those to -- x, y and z pages are really important.

Alternatively, I'd love to see Google or someone create a spidering tool that really does grab all your URLs that are spiderable, so you could easily add more if you don't have access to python or some type of programmer. I'm thinking about the smaller site owners who still have a lot of content but who can't program. And the blogging tool whipped up above sounds great.

Really interesting to see what others may do, as well. If I had to start, I'd be looking at log data and taking my most popular pages and ensuring they are being refreshed regularly and marked for top priority. Even better if my tool could make a file like this for me, for export.

[Joey]

Originally Posted by dyn4mik3

I have a concern:

How does Google feel about being "pinged" on every site update? There is no mention if there is a penalty if we resubmit our sitemap too often.

It would be real easy to write some code that "pings" Google with a new sitemap every single time a new page is created. I know that the crawler comes by in intervals, but will there be a restriction as to how often we can resubmit?

QUESTION 1 ^^
important question, i just wrote a sitemap for my forum, but there will be an update every few seconds, so what does google think if i ping them for an update of my xml file every few minutes / seconds... what do you prefer?
can't you automatically ask my sitemap if it's updated before you let your spider squeeze my site?

QUESTION 2 ^^
what are you doing against doorwaynetworks with thousands of sites?
it's much easier now to add suche spammer sites to your crawler timetable... or much easier to let you know - i don't think you have great ideas against it... - have you ever noticed GoogleAdsense stuffed content networks who only exists to spider other networks? or these wikipedia clones ... they now only need to add there complete url+keyword database to you... and earn money with your adsense programm...

[sootledir]

Great points, Danny.

It would be nice to be able to paste a URL in the search box and see details about the page. Things such as date first visited, spider frequency, etc. Nothing too controversial, but something helpful.

[rustybrick]

Originally Posted by DaveN

will get our "Inhouse Customer" CMS system to auto create the XML next week.

As will I, for all our custom CMSs.

But they are all pretty SEF, so not sure how much it will help. Crawl Frequency...New pages...

You and I have other solutions for this, outside of Google Sitemaps.

[krisval]

GG or SitemapsAdvisor,

How should we treat subdomains? Should we create a different site map for each subdomain or a single sitemap on the main domain?

ex. Should searchenginewatch.com have a different sitemap than forums.searchenginewatch.com

Thanks! And very much thanks for all of the Info.

[andrewgoodman]

Hi GG,

Very interesting and frankly shocking initiative. This should be a huge stride towards increasing indexability on demand, for any site owner that wants to take the trouble.

Please accept a silly question from a technically-challenged PPC junkie, but I have skimmed through some of the FAQ's and see the logic behind having multiple site maps (some of which may be referring to URL's in subdirectories), and only allowing URL's under those subdirectories to be included in that sitemap file.

More generally though, for smaller sites of 1,000 pages or less, is there any benefit to using multiple sitemaps or is it OK to dump a bunch of URL's into a single sitemap referring to the whole site?

Kudos on the innovation, Google.

Hello. I thought you might want to know about my horrific experiences with Google Sitemap. It's BRAND NEW and I was -- apparently -- the very first person to download their python sitemap generator from SourceForge.

Within five minutes it crashed my development server, a 3200 MHz Pentium 4 with 2GB of RAM running Debian Linux. Just imagine if this had been the production server...the costs for over-utilizng the webserver

For the details, see http://www.incendiary.ws/node/94 Please syndicate my content if you want :-)

[GoogleGuy]

In case SitemapsAdvisor is asleep, I'll try to take a couple. Marketing Guy, I think if anyone produces code for something like Wordpress or phpBB, that would be a great opportunity for us to collect several of those examples and point to them in a Google Blog post. I'll pass that suggestion on to folks. Ultimately, most people won't care if Google writes code for vbulletin or Miva (or whatever) or someone else does; as long as there's code that they can use. I wouldn't be surprised if various content management software makers provide this as a nice extra feature, for example. Or in theory a web hosting company could offer this as a value-add service. I'm just riffing though..

[GoogleGuy]

HopeSeekr of xMule, I'll pass on the feedback. I believe many people have run this internally without problems though. Do you have an unusual setup or quota on your webserver? The format is pretty well-defined, so you don't have to rely on our code to generate a sitemap if you'd rather do it yourself; I think that's what dyn4mik3 did. Or you could start by making one by hand with just a couple page on it, if you want to take Sitemaps for a test drive without running any code. But I'll ask what the expected swap footprint of the python code can be.

[Joey]

hey gg,

i am sorry about my questions.. some posts bevor;
i know it's really difficult for you guys to answer on such questions ... but may be you'va got an answer for the first one...

atm: Google Sitemaps ist down:

Server Error
The server encountered a temporary error and could not complete your request.
Please try again in 30 seconds.

[GoogleGuy]

krisval and andrewgoodman, I'll take those two questions.

It's true that you could do a different sitemap for each subdomain, or for different subdirectories. But if the site has less than 1000 pages, I would lean toward one sitemap at the root of the tree. I think the design is pretty general. That allows someone on an ISP or freehost to list their pages even if they don't have access to the root of a directory (a potential weakness of robots.txt is that it has to be at the root level).

But just because that flexibility exists doesn't mean that you have to use it, and in fact I would recommend keeping it simple for the first few days or first few times that you experiment with Sitemaps. So I'd recommend using one Sitemap for the time being, unless you just love to be on the bleeding edge.

[rustybrick]

GoogleGuy,

Are you worried about crafty Webmasters using this to some how, "unethically", spam the index?

If so, which methods do you think they will try?

Seriously, I was shocked as well. But I think its a great sign of acceptance of the "two worlds". In a way, you are kind of giving non SEOs an easy way into the index. Just one more tool to avoid the SEO consultant. Let me clarify, an SEO is still needed to "optimize" the page, but no longer make the pages "SE Friendly". In a sense....

[GoogleGuy]

That's funny, Joey! As soon as you said that, I told myself--"it must have gotten Slashdotted." Sure enough, it's the top story on http://slashdot.org/ right now. The info about Sitemaps may be running on a "normal" webserver instead of our custom setup. I'd wait for the stampede-o-technical-folks to subside; maybe read the Slashdot thread or give it a few minutes up to a half hour for the geek-slagging to stop.

I alerted the Sitemaps team, but a Slashdotting is hard enough for a regular webserver, let alone one that's doing https. I'd give it a few minutes.

[rustybrick]

On the updated Advanced Google Guidelines page, you explain "Why is my site labeled "Supplemental"?"

Supplemental sites are part of Google's auxiliary index. We're able to place fewer restraints on sites that we crawl for this supplemental index than we do on sites that are crawled for our main index. For example, the number of parameters in a URL might exclude a site from being crawled for inclusion in our main index; however, it could still be crawled and added to our supplemental index.

The index in which a site is included is completely automated; there's no way for you to select or change the index in which your site appears. Please be assured that the index in which a site is included does not affect its PageRank.

(1) Thank you for updating these pages
(2) Can submitting your site via Google Sitemaps help get a site out of the supplemental index?

[Receptional]

I am uncertain as to the motives for Google to have chosen such a complicated looking solution to achieve what they say they want to achieve. If they want a datestamp for pages, they simply ask us to datestamp pages with a metatag - no need for all this XML stuff using python surely?

This whole thing seems to have had a massive amount of development involved (though maybe not... I just got an error 502) - which looks technology led rather than market driven and I am sure the things that it says it plans to acheive could have been achieved with much less effort.

So I can only conclude that there are other objectivbes for Googke here as well - which are not so public.

[GoogleGuy]

rustybrick,
1) You're quite welcome! There's some inertia on modifying our webmaster pages when the pages are translated into lots of other languages, but it was time to reorganize the webmaster pages. I wouldn't be surprised if we stick with English for now and see if there's any extra info we want to add before it gets translated widely.
2) Potentially. I wouldn't view it as an automatic thing though.

[GoogleGuy]

I think we're pretty upfront on this one, Receptional. Personally, I had never met an XML file I did like (ha ha!). "Why all these DTDs and labels and tags?" I would say when someone presented me with an XML file--you know, just walking down the hall and someone thrusts an XML file in front of you? I hate when that happens. "What's wrong with a tab-separated text file?" I would say when the Foo group wanted to export their Bar data. But: take a look at dyn4mik3's sitemap if you get a chance. It's actually pretty clear, understandable, and shouldn't be hard to generate or process.