For months newspaper publishers have been complaining about Google, creators of the ubiquitous search engine and the largest online ad-space sellers. Publishers have argued that online news aggregators like Google News unfairly generate tons of ad revenue for Google without compensating the papers who provide its content. In a move likely to anger European publishers, Google has essentially thumbed its nose at them in a recent blog post by saying “if you don’t want to be on Google News then go ahead and block us.”
In June, the European Publishing commission delivered the Hamburg Declaration Regarding Intellectual Property Rights to the European Commission. The declaration has 169 signatories, most of which are German publishers but also a few international heavy hitters such as Dow Jones and Newscorp. The publishers advocate that they should be able to control who has access to their sites and what they can do with them. This comes in the wake of numerous grumblings in Europe and elsewhere concerning Google News and similar aggregators. News aggregators typically don't create any original content themselves but rather crawl the web looking for stories to link to. Google then generates revenue by harvesting information and selling targeted online ad space on its search pages, none of which it shares with the papers it links to. Publishers argue this is creating a music industry-like problem where people get their content for free and only the search engines profit. Furthermore, with a near-monopoly in the online ad-space market Google could theoretically give preferential treatment to the stories and sites that generate the most revenue.
Google responded to the Hamburg Declaration with a post on its own European Public Policy Blog. The post points out that there is a simple technical fix that has been around for years to stop search engines from indexing pages: The “robots.txt” method. Webmasters simply have to insert two short lines of code into their sites in order to block web-crawling robots from searching them. The post goes on to criticize the publishers’ proposals, saying that they would “fundamentally change – for the worse – The way the web works.”
So, if there’s such a simple method for blocking Google and its ilk, why are the publishers so upset? It really comes down to them wanting to enjoy the best of both worlds. On the one hand, they want (or perhaps need) to be indexed by search engines. Millions of people get their news via aggregators, driving up the publishers’ own traffic and readership. Yet, on the other hand they want Google to pay royalties for the privilege of linking to their sites. The publishers don’t want to block Google, they want Google to start coughing up.
Whether the publishers realize it or not, what they are asking for would indeed change the web for the worse. It would transform Google's role from that of an open, automated search engine into a discerning news distributor. Furthermore, the implications it has for the rest of the net are severe. Should every website have to pay the target site whenever they post a link? Sorry newspaper publishers, but I’m with Google on this one: If you want your sites to be publicly accessible you need to take the good with the bad.