Google Expands Robots Exclusion Protocol, Unavailable After Tag Now Live

Jul 30, 2007 - 9:22 am 0 — by

As we've discussed before, Google has been planning an unavailable_after tag that would enable webmasters to alert Google to when content should no longer be crawled by the Googlebot. A Google Blog post by Dan Crow announces that this feature is now live. Additionally, Google is now accepting a META tag to be associated with any file. This allows webmasters to control access to PDFs, graphics, video, and audio files.

WebmasterWorld members are glad that Google is embracing additions to the robots.txt protocol, but they're concerned that other search engines won't follow suit. This is great for Google, but webmasters still need to be in control over the META elements as before for Yahoo, Ask, and MSN.

Barry over at Search Engine Land also announces the news, and beu at Search Engine Watch Forums alerts the community of this addendum as well.

Forum discussion at WebmasterWorld and Search Engine Watch Forums.

 

Popular Categories

The Pulse of the search community

Google Search Volatility

More Details

Search Video Recaps

 
Video Details More Videos Subscribe to Videos

Most Recent Articles

Search Forum Recap

Daily Search Forum Recap: November 20, 2026

Nov 20, 2026 - 10:00 am
Search Video Recaps

Search News Buzz Video Recap: Google November Core Update Status, FBI Alert On SEO Poisoning, Bing & ChatGPT Web Only Links, Microsoft Ads Bid Upgrade

Nov 20, 2026 - 8:01 am
Google

Google, ChatGPT & Others AI Engines Provide Coupon Codes

Nov 20, 2026 - 7:51 am
ChatGPT

ChatGPT Card Design

Nov 20, 2026 - 7:41 am
Google Ads

Google Expands & Clarifies Sexual Content Advertising Policy Examples

Nov 20, 2026 - 7:31 am
Bing Ads

Bing & Microsoft Ads - Made With AI Label

Nov 20, 2026 - 7:21 am
Previous Story: Wikipedia Founder Proposes Open Source Search Engine