Anatomy of Large-Scale Digital Technology

A couple of very interesting research papers that I came across recently…both are looking at cross sections of complex data systems and how they interact and operate. One of these is old (in technology terms), from 1998 and describes the emergence of Google;

Google utilizes link to improve search results.  The citation (link) graph of the web is an important resource that has largely gone unused in existing web search engines. We have created maps containing as many as 518 million of these hyperlinks, a significant sample of the total. These maps allow rapid calculation of a web page’s “PageRank“, an objective measure of its citation importance that corresponds well with people’s subjective idea of importance.

When you read this now, it seems a pretty basic idea – rank webpages based on how many other webpages link to it as a source. You are using the wisdom of the crowds to do the work – trusting that if millions and millions of web-users think a site is important; it must be. This had a number of advantages over how people were doing search prior to this. First it allowed the system to scale with the proliferation of users and sites. So as more users arrived, hey created more sites which had links that ranked sites and so on in a positive feedback loop that already existed. Google was like the creator of a microscpoe peering into water to discover a whole new world of micro-organisms/data that we could not see before.

Second is paper inspired by this; looking at how social networks influence search and information.  This paper follows on the the idea of PageRanks but moves it deeper into the interconnections we have created since 1998;

We demonstrate that there is a large class of subjective questions — especially longer, contextualized requests for recommendations or advice — which are better served by social search than by web search. And our key finding is that whereas in the Library paradigm, users trust information depending upon the authority of its author, in the Village paradigm, trust comes from our sense of intimacy and connection with the person we are getting an answer from.

Both are worth a read. (Also interesting to note that the company founded by the authors of the paper has been acquired by Google!)

Hat-tip to Peter for the links.