Daily Telegraph Wants to Sue Google and Yahoo for Crawling its Site

Apr 25, 2007 • 11:20 am | comments (6) by twitter | Filed Under Google Search Engine

It seems that a lot of well-known companies have webmasters (or legal departments) who just don't have a clue how to implement a robots.txt file. According to a DigitalPoint Forums thread, the United Kingdom based Daily Telegraph is looking to sue Google and Yahoo for accessing its content.

Their statement, as quoted in the Guardian Unlimited, is that they are concerned that these search engines are accessing content for free and don't give them proper credit.

Our ability to protect content is under consistent attack from those such as Google and Yahoo who wish to access it for free. These companies are seeking to build a business model on the back of our own investment without recognition. All media companies need to be on guard for this. Success in the digital age, as we have seen in our own company, is going to require massive investment... [this needs] effective legal protection for our content, in such a way that allows us to invest for the future.

Apparently, they're clueless about implementing a robots.txt file that will prevent search engines from accessing content "for free." As of this writing, this is its current robots.txt file:

# Robots.txt file # All robots will spider the domain

User-agent: *

Disallow: */ixale/

Not only that, but they have the ability to remove content from the SERPs in Google and in Yahoo.

It is a bit disturbing how many people are concerned about search engines (which ultimately give them more visibility!) The claim that search engines don't respect their rules goes both ways. Daily Telegraph, I imagine you have rules you want Google and Yahoo to respect. Well, the search engines have rules too. Follow them and you'll be fine.

Feel free to add your two cents on the DigitalPoint Forums thread.

Previous story: Do Irrelevant Google AdSense Ads Frustrate You?


Brian Rants

04/25/2007 05:19 pm

Wow. This is wrong on so many levels. Not only do they have control over robots with robots.txt, but they should publish how much traffic they get from Google and Yahoo. My guess is they know exactly how much traffic they get, and they know all about using robots.txt. They want the visibility of Google and Yahoo without the indexing of the search engines (alleged copyright infringement). Uhhh...you can't have it both ways.

SEO Egghead

04/26/2007 12:18 am

I'm not sure they're wrong according to the letter of the law -- but that's very old news. Fact is, according to my interpretation of copyright law, unless you explicitly grant a license, it's infringement. It's NOT the opposite way around. Sorry. It's not public domain if you don't say otherwise. That responsibility is not yours. Thing is, normatively, it's become acceptable, and it's not like I really agree with TDT, but you see the point ....

Mark Hammersley

04/26/2007 10:18 am

The Daily Telegraph is a huge newspaper here in the UK. It seems odd that they would make this kind of statement without consulting anyone who knows about robots.txt files and alike. That said it does highlight that some major media companies just do not get the web. I know they think they are standing up for THE MEDIA but they need to realise that their business model needs to completely change.


04/26/2007 10:41 am

I’m no expert but relating to the copyright comment by SEO egghead, is this not nullified by the Digital millennium act whereby this flips the rules and content can be referenced as long as its cited correctly and a specific request has been made by a company NOT to reference it. Like I said I could be wrong...


04/26/2007 03:31 pm

How ridiculous. Are they saying they don't WANT that traffic that search engines send? AND, they must not have an understanding of robots.txt, because it's free range for search engines on the site. Sounds like someone "important" over there (and by important, I mean they yell a lot so that people can hear them while their head is up their a**) looked at lower circulation numbers and got upset.


09/12/2007 02:14 am

I cant see the point in all this I WANT every search engine in the world to see my site.people PAY a fortune to get sites listed.Is this not the point of having a site so people see it? If you don't want people to see something do not put on site anywhere or encrypt it,simple so come and have a look!

blog comments powered by Disqus