Googlebot Now Crawling Google News

Google News recently made a new update to crawl the news using Googlebot. Well, it might not be a very important update for most of the publishers and thus, news organizations who wish to opt out of Google News can do that. But I’m sure that none of them would want to do that considering the reach of Google News network.

Google Help Center has provided some guidelines on how to use robots protocol if you wish to be excluded or included. Some of the main points are listed here:

• You will only see Googlebot user-agent in your website’s traffic logs. Google has clarified though that Googlebot instead of Googlebot-News does not matter as long as inclusion policies are concerned. So, if you opted out of Googlebot crawling your news and you see Googlebot coming in your traffic logs, don’t panic. Google respects your decision but is just trying to make the naming conventions standard. Moreover, if you still have doubts, you can check your website in Google News search bar by typing, site:abc.com.
• If you are using any analytics tool, then you should be able to see two sets of traffic just like before, one coming from Google Search and one coming from Google News. The only difference is that the automated visits from Googlebot-news crawler will not longer be visible.
• If your website is complying to the guidelines for Googlebot then you don’t have to make any changes to your website. Though the websites that require login or registration before displaying the full article, Google News will only be able to crawl the text that is visible prior to login.
• The News Sitemap of websites will still be crawled in the normal way. So, the news publishers who have not setup a news sitemap, this is a good wakeup call for you guys to make one.

To summarize in simple words, any news publishers who wish to opt out of Google News and want to stay in Google Search can disallow the news bot and allow the search bot. You can check out the help article on how to do this here.

From SEO or Marketing perspectives, I’m really not sure how this update would help, but let’s see how many news websites go for the change.