Yahoo! Slurping Wildcards Via Robots.txt File

Nov 6, 2006 - 7:14 am 2 by
Filed Under Yahoo SEO

Thursday night, last week, the Yahoo! Search Blog wrote Yahoo! Search Crawler (Yahoo! Slurp) - Supporting wildcards in robots.txt. I am honestly a bit shocked by the SEO community's response to this, or lack thereof.

I have spotted two threads on the topic, both just threads that say, that Yahoo! announced it and nothing more. I know this is something Webmasters have requested in the past and that they have looked down upon Yahoo! for not supporting until now.

The two additional parameters include:

You can now use '*' in robots directives for Yahoo! Slurp to wildcard match a sequence of characters in your URL.
You can now also use '$' in robots directives for Slurp to anchor the match to the end of the URL string.

More details at

Forum discussion at WebmasterWorld & Search Engine Watch Forums.


Popular Categories

The Pulse of the search community


Search Video Recaps

Video Details More Videos Subscribe to Videos

Most Recent Articles

Search Forum Recap

Daily Search Forum Recap: May 27, 2024

May 27, 2024 - 10:00 am

In Face Of AI Overview Backlash, Google Updates Docs With How To Show Web Only Results & How To Give Feedback

May 27, 2024 - 7:51 am
Google Search Engine Optimization

Google's John Mueller Blasts The Concept Of Toxic Links, Again

May 27, 2024 - 7:41 am
Google Search Engine Optimization

Some Reporting Fewer Links Reported In Google Search Console

May 27, 2024 - 7:31 am

Google Images "See Exact Matches" Helps You Find Who Stole Your Images

May 27, 2024 - 7:21 am

Google Mobile Tests Large Blue Visit Button On Search Results

May 27, 2024 - 7:11 am
Previous Story: Running Out of AdWords Campaigns? Got Some Tips For You