Google's Crawler Only Submit Forms When It Can't Find Your Content

Sep 15, 2015 - 8:32 am 0 by

Keyboard1 Google 1900px

Google's spider, GoogleBot, has really been getting clever over the years. Crawling JavaScript, accessing image data, finding links to consume in places you can't see them and more. But GoogleBot has also been spotted submitting forms.

Imagine, GoogleBot going to Bing.com, entering in a query and then crawling the web through links BingBot discovered. Oh wait, Bing did that - but Google does that on some sites.

John Mueller of Google answered the question "does Google submit forms?" on a Google+ Hangout at the 29:10 mark. John said they do so in very specific situations. He said:

We do that in some very specific situations where we think that this looks like a search form and we are not getting all of the content that we suspect is available on this web site. Then in those kind of situations, maybe we will try to plug in some keywords from the existing content to see if we can find more data and pages. But that is the really rare situation. It is usually not the case that we go to a site and so oh, there is a form from here, let me enter some gibberish and click submit and see what happens.

In practice, those forms don’t lead us to new or interesting content. Search forms sometimes do lead us to content that we can’t otherwise find, especially if a site doesn’t have any navigation, doesn’t have a sitemap file. If we see that maybe the home page just has a search form, and we suspect there is a lot of really good content there, but we can’t really reach it in our crawl, so that is one situation where our algorithms might say well, we know this site is about this topic, lets pug in a keyword and see what we find.

This has been going on for some time but it is good to hear why Google does it and when Google does it.

Forum discussion at Google+.

This post was pre-written and scheduled to be posted today. Have a happy, healthy and sweet new year!

 

Popular Categories

The Pulse of the search community

Follow

Search Video Recaps

 
Google Core Update Rumbling, Manual Actions FAQs, Core Web Vitals Updates, AI, Bing, Ads & More - YouTube
Video Details More Videos Subscribe to Videos

Most Recent Articles

Search Forum Recap

Daily Search Forum Recap: March 18, 2024

Mar 18, 2024 - 4:00 pm
Google Updates

Google Urges Patience As The March 2024 Core Update Continues To Rollout

Mar 18, 2024 - 7:51 am
Google

Official: Google Replaces Perspective Filter With Forums Filter

Mar 18, 2024 - 7:41 am
Google Maps

Google Business Profiles Now Offers Additional Review After Appeal Is Denied

Mar 18, 2024 - 7:31 am
Google Maps

EU Searchers Complaining About Google Maps Features Changes Related To DMA

Mar 18, 2024 - 7:21 am
Google

Google Showing Fewer Sitelinks Within Search

Mar 18, 2024 - 7:11 am
Previous Story: Thumbtack Risking A New Google Link Penalty?