Google: Disallow Crawling Of Action URLs (Buy Now, Sign Up, etc)

Jun 13, 2024 - 7:31 am 0 by

Google Robot With Credit Card

Google posted a public service announcement saying you should disallow Googlebot from crawling your action URLs. Gary Illyes from Google posted on LinkedIn, "You should really disallow crawling of your action URLs. Crawlers will not buy that organic non-GMO scented candle, nor do they care for a wishlist."

I mean, this is not new advice. Why let a spider crawl pages where it cannot really take any actions. Googlebot can not make purchases, cannot sign up for your newsletter, and so on.

Gary wrote:

A common complaint we get about crawling is that were crawling too much, which uses too much of the server's resources (though doesn't cause problems otherwise). Looking at what we're crawling from the sites in the complaints, way too often it's action URLs such as "add to cart" and "add to wishlist". These are useless for crawlers and you likely don't want them to be crawled.

If you have URLs like:

https://example․com/product/scented-candle-v1?add_to_cart

and

https://example․com/product/scented-candle-v1?add_to_wishlist

How should you block Googlebot? He said, "You should probably add a disallow rule for them in your robots.txt file. Converting them to HTTP POST method also works, though many crawlers can and will make POST requests, so keep that in mind."

Now, a few years ago, we reported that Googlebot can add products to your cart to verify your pricing is correct. It seems to be part of the merchant shopping experience score feature - so I'd be a tad careful with all of this.

Forum discussion at LinkedIn.

Note: This was pre-written and scheduled to be posted today, I am currently offline for Shavout.

 

Popular Categories

The Pulse of the search community

Follow

Search Video Recaps

 
Google Volatility, Indexing Bugs, Google Ads Broad Match Default & YouTube SEO Tips - YouTube
Video Details More Videos Subscribe to Videos

Most Recent Articles

Search Forum Recap

Daily Search Forum Recap: July 12, 2024

Jul 12, 2024 - 10:00 am
Search Video Recaps

Search News Buzz Video Recap: Google Volatility, Indexing Bugs, Google Ads Broad Match Default & YouTube SEO Tips

Jul 12, 2024 - 8:01 am
Google Maps

Remove Thumbtack Book Links In Google Business Profiles

Jul 12, 2024 - 7:51 am
Google Search Engine Optimization

New: Google Search Console Let's You Add Your Shipping & Return Information

Jul 12, 2024 - 7:41 am
Google

Google Search Tests Autoplaying Video Thumbnails In Results?

Jul 12, 2024 - 7:31 am
Bing Search

Bing Search Tests Voting On Search Results

Jul 12, 2024 - 7:21 am
Previous Story: Bing Tests Black Review Stars In Search Results