"Time to Refresh" a Webpage on the Search Engine's Index

[Nacho]

After reading Garrett French’s article on “Cache Bashing (http://www.webpronews.com/insiderreports/searchinsider/wpn-49-20040521CacheBashingGoogleAPIUsedInSEOWar.html)” (a great one!) it got me thinking on the idea of TIME TO REFRESH from the search engines index. For example, I notice that for most important news sites refresh is about 24 to 48 hrs to refresh on the homepage (maybe faster on the BIG brands), however an above average retail .com site might take about 4 to 6 days to refresh.

What does it take to increase the “time to refresh” or make a webpage more sticky for the web spider to want to come back more often and re-index? I think this is a topic that will continue to grow, especially when companies like Yahoo! want to charge a fee (via “Site Match”) because they are using more company resources to get that spider to refresh a web page more often. Will this be the future of “time to refresh” or will website owners and search engine spiders figure out a balance to get your website updated on the search engine’s index as fast as it is necessary?

Please share your thoughts on this topic and give examples of techniques to increase “time to refresh”.

[rustybrick]

About a year ago it was fairly simple with Google. If your PageRank was X then "freshbot" came to visit you every day, if it was Y then freshbot came Y days, etc. That is why, a year ago, everyone was crazy about the ~monthly deepcrawl, where there news sites were updated completely and PageRank was updated.

Today, its not that simple.

Basically, the search engines look at how often your pages are updated. If they see your pages are updated daily, then they will make an effort to visit your pages every 24 - 48 hours. If you update weekly, then the search engines will visit weekly. They try to look for patterns in how often a page is updated.

[Nacho]

** BUMP to get this topic back rolling **

Actually with Google and MSN bots running all over the place like chickens without a head, we see them back and back again almost every day including deep crawls.

However, what I very interested in discussing here are methods on how to train the bots to crawl your pages more and more often. For example:

Add a news section or fresh content that rotates in and then archives.
Add new products to your category pages as maybe "featured" products.
Etc.

[rustybrick]

Wouldn't a time stamp, RSS feed, dynamic rotation of content, and the like all provide sufficient reason for the bots to check back? :)

[creativecraig]

Adding new content (news) to the home page is ample enough for the spiders to come visiting to my sites every day.

RSS news feeds work a treat as well, I know this from experience - industry related news fed to your home page 3 or four times a day and then archived in a search engine friendly way can add relevant content and will keep the spiders visiting daily.