Google: Providing APIs Won't Reduce Search Results Scraping

Jun 29, 2018 - 7:21 am 4 by

Google Scrape

There was a somewhat fun conversation on Twitter around the new URL inspector tool. Some SEOs are saying that if Google provided an API for this tool, it would reduce scraping of the Google search results. As you know, scraping Google's search results is against their terms of service and Google is not a fan of those that do it.

But the truth is, Google has done very little to stop the popular tools from scraping their search results. There have been times where they caused problems for the tools but overall, scrapers still do what they do best, scrape.

Now when someone mentioned to John Mueller of Google that providing this API for the URL inspector tool would reduce scraping, John laughed. He said, "Hah, I doubt it." Adding also that it is not right to say we will break your TOS (terms of service) if you don't give us a feature we want, is not a good argument.

The topic of scraping is very interesting because technically Google is one big scraper but of course, we can always use robots.txt or other methods to prevent Google from indexing or scraping our content.

Forum discussion at Twitter.

 

Popular Categories

The Pulse of the search community

Search Video Recaps

 
- YouTube
Video Details More Videos Subscribe to Videos

Most Recent Articles

Search Forum Recap

Daily Search Forum Recap: May 15, 2025

May 15, 2025 - 10:00 am
Google

Google Rethinking Search Stack From Ground Up With LLMs

May 15, 2025 - 7:51 am
Google Search Engine Optimization

Google Doc: Link Position & Click Bias Ranking Bias Adjustment

May 15, 2025 - 7:41 am
Google Ads

Google Ads New "Ads Funded By" Details

May 15, 2025 - 7:31 am
Google Ads

Google Ads Device Targeting Controls For Performance Max Campaigns

May 15, 2025 - 7:21 am
Google

Google Search Tests Vertical Search Labels In Autocomplete

May 15, 2025 - 7:11 am
Previous Story: Google: Google Analytics Won't Hurt Or Benefit Your Search Rankings