A new bill in the UK Parliament named the Digital Economy Bill [HL] 2009-10 is proposing to give search engines, such as Google, a form of immunity against being sued over copyright infringement. It is a bit more complex than that but overall, if you want your content out of the search engines, block them - otherwise, you can't sue them over copyright law.
A Sphinn thread is pretty heated over the topic. Primarily between Michael Gray and Danny Sullivan. Let me quote some of the conversation:
Danny Sullivan in response to Michael Gray:
Yeah, yeah, simmer down there troll boy :)
So the actual article this is talking about from The Guardian says this proposal also says:
The presumption (of having an automatic license) may be rebutted by explicit evidence that such a licence was not granted. Such explicit evidence shall be found only in the form of statements in a machine-readable file to be placed on the website and accessible to providers of search engine services.
In other words, this gives robots.txt legal backing. You block that way, search engines can't index you. Fair enough. I mean, that's how things have worked for ages with the respected search engines. But if some rogue spider copied you, you couldn't easily claim a copyright violation because robots.txt had no force of law. Now, you could sue saying they'd been restricted and still indexed your content.
Michael Gray in response to Danny Sullivan:
being a troll boy ;-) and not a lawyer I may be missing something, but this seems pretty clear...
In other words, Google would be free to copy everything - but a publisherblocking search spiders with a robots.txt file would be taken as withholding that right. An explicit "fair use" provision, which Google often cites against copyright-abuse claims, does not exist in UK law.
Google can copy whatever it wants, unless you block it with robots, so if you want to retain you copyright then you do so by slitting your own throat for search engine traffic. That just doesn't make any sense for anyone ... except google.
The debate goes on and on in the thread, so if you are in a troll/rant mood or if you just find the topic interesting, do check it out.
Forum discussion at Sphinn.