Google: Two Common Reasons When A Spike In Crawling Is Bad

Jun 19, 2024 - 7:31 am 1 by

Google Server Explosion

Google's Gary Illyes posted on LinkedIn with two common examples of when a spike in Googlebot activity, crawling, is a bad thing. The short answer is when Googlebot gets to crawling an infinite section of your site (like calendar pages that goes on forever) and when your site is hacked with a ton of new hacked pages.

This is probably very basic SEO stuff for most of you but there might be a reason he is sharing this now.

Gary wrote, "Don't get happy prematurely when search engines unexpectedly start to crawl like crazy from your site." "A sudden increase in crawling can mean good things, sure, but it can also mean something is wrong," he added.

In these cases, he said, "Treat unexpected sharp increases in crawling as a symptom of an issue until you can prove otherwise." Then he joked, "Or, you know, maybe I'm just a hardline pessimist."

Here are two issues that come up way too often when looking at sharp increases in crawling, he wrote:

Infinite Spaces:

The example on infinite spaces that he provided is one of the most common. When "you have a calendar thingie on your site, or an infinitely filterable product listings page. If your site generally has pages that search users find helpful, crawlers will get excited about these infinite spaces for a time. robots.txt is your friend, use it."

Hacked Content:

Another common example is hacked content. He said, "if a no-good-doer somehow managed to get access to your server's file system or your content management system, they might flood your otherwise dandy site with, well, crap. If your site generally has pages that search users find helpful, crawlers will get excited about these new pages for a time and happily crawl them. https://web.dev/hacked has great resources about what to do in these cases. (tangent: yes, this is more cracking than hacking but apparently the internet is fine with the misnomer)."

Forum discussion at LinkedIn.

 

Popular Categories

The Pulse of the search community

Follow

Search Video Recaps

 
Google Volatility, Indexing Bugs, Google Ads Broad Match Default & YouTube SEO Tips - YouTube
Video Details More Videos Subscribe to Videos

Most Recent Articles

Search Forum Recap

Daily Search Forum Recap: July 12, 2024

Jul 12, 2024 - 10:00 am
Search Video Recaps

Search News Buzz Video Recap: Google Volatility, Indexing Bugs, Google Ads Broad Match Default & YouTube SEO Tips

Jul 12, 2024 - 8:01 am
Google Maps

Remove Thumbtack Book Links In Google Business Profiles

Jul 12, 2024 - 7:51 am
Google Search Engine Optimization

New: Google Search Console Let's You Add Your Shipping & Return Information

Jul 12, 2024 - 7:41 am
Google

Google Search Tests Autoplaying Video Thumbnails In Results?

Jul 12, 2024 - 7:31 am
Bing Search

Bing Search Tests Voting On Search Results

Jul 12, 2024 - 7:21 am
Previous Story: Google Tests Two More People Also Ask Results (6 PPA)