IndustryGoogle News Content Now Crawled via Googlebot

Google News Content Now Crawled via Googlebot

Due to an infrastructure change, Google News is now using the Googlebot instead of a dedicated crawler to examine web content. While most elements remain the same, webmasters should be aware of some minor differences in indexing and crawling.

Google News is now using the Googlebot instead of a dedicated crawler to examine web content. While most elements of crawling remain the same, webmasters should be aware of some minor differences.

Changes with Googlebot Crawling

google-newsAs announced on the Official Google Webmaster blog, all content for Google News will now be crawled by the standard Googlebot starting immediately. The change is presumably in effect on a global scale.

How should your behavior change? That really depends how you have your site set up currently.

For the most part, the Googlebot has been set to behave identically to the news crawler when it comes to news content; a negative “Googlebot-News” entry in the robots.txt file will still halt just the news crawling, sitemaps will still be crawled, and all analytics for actual visitors will remain the same.

There are a couple differences, however.

First, when you examine your site logs, you’ll only see the Googlebot user-agent. This makes it slightly harder to know if your site is being indexed and included in Google News. Google suggests that “You can always check whether your site is included in Google News by searching with the ‘site:’ operator.” To do this, simply go to news.google.com and enter “site:yoursite.com,” conduct the search, and see if you get any results. If you do, then your site is being indexed for news content.

Additionally, the guidelines for Googlebot are now fully applicable for news content. That means that any pages that require payment or login prior to being viewable will not be fully indexed. Rather, only the title and user-visible snippet will be seen by Google’s crawler. Google has offered some information and potential solutions for subscription publishers via a webmaster help article.

By and large, this is just a change to Google’s back-end. Your basic algorithm ranking won’t change, non-inclusion will still be respected, and all your content will go on basically as it did before. However, knowing the details of these minor changes is likely to be of some help for news sites.

Resources

The 2023 B2B Superpowers Index
whitepaper | Analytics

The 2023 B2B Superpowers Index

8m
Data Analytics in Marketing
whitepaper | Analytics

Data Analytics in Marketing

10m
The Third-Party Data Deprecation Playbook
whitepaper | Digital Marketing

The Third-Party Data Deprecation Playbook

1y
Utilizing Email To Stop Fraud-eCommerce Client Fraud Case Study
whitepaper | Digital Marketing

Utilizing Email To Stop Fraud-eCommerce Client Fraud Case Study

1y