With inimitable clarity, Cory Doctorow made the case for an open alternative to Google in The Guardian earlier this month. He focused on the secrecy of search:
[S]earch engines routinely disappear websites for violating unpublished, invisible rules. Many of these sites are spammers, link-farmers, malware sneezers and other gamers of the system. . . . The stakes for search-engine placement are so high that it’s inevitable that some people will try anything to get the right placement for their products, services, ideas and agendas. Hence the search engine’s prerogative of enforcing the death penalty on sites that undermine the quality of search.
[Nevertheless, i]t’s a terrible idea to vest this much power with one company, even one as fun, user-centered and technologically excellent as Google. It’s too much power for a handful of companies to wield.
Search engines like Google have some good reasons for keeping their algorithms confidential–if they were public, manipulators could quickly swamp Google users with irrelevant results. However, just as Comcast cannot circumvent net neutrality regulation by saying all its traffic management and spam-fighting methods are trade secrets, search engines should not be able to use such arguments to escape regulation altogether. Moreover, there are ways of developing a qualified transparency that would let a trusted third party examine a search engine’s conduct without exposing its business methods for all the world to see.
But Doctorow does not want regulation here–he wants an alternative. Having made a similar case for a “public option” in the case of health insurance, I like this line of argument, but I think Doctorow is underestimating the barriers to entry. Though he’s aware of the failure of Wikia, Doctorow wonders if a “wikipedia for search” could be built:
We can imagine a public, open process to write search engine ranking systems, crawlers and the other minutiae. But can an ad-hoc group of net-heads marshall the server resources to store copies of the entire Internet? . . . . It would require vast resources. But it would have one gigantic advantage over the proprietary search engines: rather than relying on weak “security through obscurity” to fight spammers, creeps and parasites, such a system could exploit the powerful principles of peer review that are the gold standard in all other areas of information security.
The “rival public system” approach has been suggested for search engines a few times before. About a decade ago, Introna & Nissenbaum demonstrated that “the conditions needed for a marketplace to function in a ‘democratic’ and efficient way are simply not met in the case of search engines.” Recognizing this, Jean-Noel Jeanneny made a case for a French language alternative to dominant US-based search engines. The Quaero project in the EU appears to be answering that call, though in a far more dirigiste manner than Doctorow would probably like.