Google's spider, GoogleBot, has really been getting clever over the years. Crawling JavaScript, accessing image data, finding links to consume in places you can't see them and more. But GoogleBot has also been spotted submitting forms.
Imagine, GoogleBot going to Bing.com, entering in a query and then crawling the web through links BingBot discovered. Oh wait, Bing did that - but Google does that on some sites.
John Mueller of Google answered the question "does Google submit forms?" on a Google+ Hangout at the 29:10 mark. John said they do so in very specific situations. He said:
We do that in some very specific situations where we think that this looks like a search form and we are not getting all of the content that we suspect is available on this web site. Then in those kind of situations, maybe we will try to plug in some keywords from the existing content to see if we can find more data and pages. But that is the really rare situation. It is usually not the case that we go to a site and so oh, there is a form from here, let me enter some gibberish and click submit and see what happens.In practice, those forms don’t lead us to new or interesting content. Search forms sometimes do lead us to content that we can’t otherwise find, especially if a site doesn’t have any navigation, doesn’t have a sitemap file. If we see that maybe the home page just has a search form, and we suspect there is a lot of really good content there, but we can’t really reach it in our crawl, so that is one situation where our algorithms might say well, we know this site is about this topic, lets pug in a keyword and see what we find.
This has been going on for some time but it is good to hear why Google does it and when Google does it.
Forum discussion at Google+.
This post was pre-written and scheduled to be posted today. Have a happy, healthy and sweet new year!
