Google's Crawler Only Submit Forms When It Can't Find Your Content

Sep 15, 2015 - 8:32 am 0 by

Keyboard1 Google 1900px

Google's spider, GoogleBot, has really been getting clever over the years. Crawling JavaScript, accessing image data, finding links to consume in places you can't see them and more. But GoogleBot has also been spotted submitting forms.

Imagine, GoogleBot going to Bing.com, entering in a query and then crawling the web through links BingBot discovered. Oh wait, Bing did that - but Google does that on some sites.

John Mueller of Google answered the question "does Google submit forms?" on a Google+ Hangout at the 29:10 mark. John said they do so in very specific situations. He said:

We do that in some very specific situations where we think that this looks like a search form and we are not getting all of the content that we suspect is available on this web site. Then in those kind of situations, maybe we will try to plug in some keywords from the existing content to see if we can find more data and pages. But that is the really rare situation. It is usually not the case that we go to a site and so oh, there is a form from here, let me enter some gibberish and click submit and see what happens.

In practice, those forms don’t lead us to new or interesting content. Search forms sometimes do lead us to content that we can’t otherwise find, especially if a site doesn’t have any navigation, doesn’t have a sitemap file. If we see that maybe the home page just has a search form, and we suspect there is a lot of really good content there, but we can’t really reach it in our crawl, so that is one situation where our algorithms might say well, we know this site is about this topic, lets pug in a keyword and see what we find.

This has been going on for some time but it is good to hear why Google does it and when Google does it.

Forum discussion at Google+.

This post was pre-written and scheduled to be posted today. Have a happy, healthy and sweet new year!

 

Popular Categories

The Pulse of the search community

Follow

Search Video Recaps

 
Video Details More Videos Subscribe to Videos

Most Recent Articles

Google

Google Confirms Search Leak But Urges Caution

May 30, 2024 - 7:51 am
Google Search Engine Optimization

Google Report Spam Tool Adds Site Reputation & Expired Domains Abuse

May 30, 2024 - 7:43 am
Google Search Engine Optimization

Huge Decline In Links Reported In Google Search Console Links Report

May 30, 2024 - 7:41 am
Google

Google People Also Ask Tests Shaded Boxed Interface

May 30, 2024 - 7:31 am
Google

Google Early Adopter Badge For Using Search Notes

May 30, 2024 - 7:21 am
Google Maps

Google Maps Adds Wheelchair Accessible Icon To Business Listings

May 30, 2024 - 7:11 am
Previous Story: Thumbtack Risking A New Google Link Penalty?