Entries from Search Engine Roundtable tagged with 'robot'

Google's Video Crawler Not Respecting Robots.txt Directives?

It seems like we have confirmed reports from a Googler in Google Groups that Google's video crawler, part of the GoogleBot family, is not playing nice. In short, even though you may be telling Google not to crawl your videos,...

Premium Sponsors + advertise

To subscribe to the Search Engine Roundtable, click here