A Great Becoming…

Update!  I’ve now passed my PhD and now have a doctorate!  The full title is: Network Media Distribution Systems: Understanding the Media Ecology of Software Using an Evolutionary Framework.

Below are the slides from my presentations at eComm 2009, Virt3c@Hull 2010, Pervasive Media Lab (Bristol) and OKCon 2010 with notes below each explaining a bit more… (This research also formed the basis of a peer-reviewed article in the Journal of eLearning and Digital Media & an article on GamesIndustry.biz)

The talk is about my PhD research looking at technology and evolution.  I focus on p2p technology but think the finding do apply on a wider scale.  The seed to my research came from working on a p2p client development project.  What we noticed was that at a set point in the project life, our version of the project (which used Azureus as a base) was no longer compatible with the ongoing development of the main strand of Azureus (now called Vuse).  This stuck me as interesting – almost like the branching of two new species who can no longer inter-breed.  I thought it is worth looking at biological models, specificity evolution, to help understanding what is going on…

First it is worth exploring what evolution is, as there are lots of misconceptions about this.  The image above shows some of the famous finches from the Galapagos Islands that inspired Darwin to make the vital conceptual link to figure out what was happening in nature.  Darwin only had the end-results of evolution – living things and fossils – to look at.  He could not see the process and so had to reverse-engineer evolution from his observations.

The quote below shows what he found. He observed small differences in the same species and noted that they must somehow have a common root.  The famous example of his reasoning is the variations of finches he encountered on his voyage with the ship, HMS Beagle. Each separate species of finch has adapted to the conditions on its island and the food available there.   e.g Short beaks are good for tearing at the base of cactus and eating the pulp and insects that they find there.  Long beaks are good for punching holes in prickly-pear fruit and eating the flesh.

Pulling these strands together, we can see that evolution has three basic components;

• Mendelian Genetics; passing on your characterises to the next generation,
• Mutation; Copying errors in the generational process.  This is a means of change being introducing change to the next generation – reproduction methods also do this; sexual reproduction is a good example of this.
• Natural selection; nature selecting which individuals will survive to procreate, thus selecting the ‘fittest‘ genes for the next generation.

He then combined this idea with the fact that when living things procreate, they tend to produce more offspring than is needed to continue the species.  He noted that all the offspring seem to have tiny differences in each.  Many of the offspring will not make it to adulthood to procreate in turn.  He noted that the offspring with some kind of advantage (however small) are more likely to make it and so pass that advantage on to future generations.

Other authors are looked at the idea of applying the concept of evolution to technology. Basalla (1988) noted that the concept of the noble inventor was a myth; that great inventions were built on the back of lots of existing innovations.  For example: Invention of the telephone – Alexander Graham Bell – but it was built on the progress of other inventions; electricity, telegraph etc.

More recently W Brian Authur (2009) suggest that the dominant force of technology is the idea of ‘combinatorial evolution’ that is technology evolves by the combination of existing forms of technology.  He said that this does happen in biology, but is rare – however in technology, this is a common form of evolution.

We do see this in biology, but it is rare; example endosymbiotic theory suggests that the mitochondria (the power plant of a cell) were separate organisms that somehow   got engulfed into a cell, formed a symbiotic relationship and the rest is history (our history!).

For example – GPS – a new technology but one that consists of a number of pre-existing forms of technology; computers, satellites etc:  “A GPS system is a combination of computer processors, satellites, atomic clocks, radio transmitters and receivers. Whoever invented that did not say, “I am going to combine existing technologies”; they’re basically saying, “What is it going to take to solve a problem here—the one of finding a point’s location on the earth?” And that combination resulted. So technology evolves by combination, and once the technology’s in place, then the Darwinian mechanisms of variation and selection set in.”

However Arthur does not seem to account for where we see smaller incremental improvements in a technology.  If as an example we look at the technology changes is modems:

• 1980s – Used  frequency-shift keying as a method of turning sound into digital and back again.
• Early 1990s – Based on the same idea but added 90 degree variation of frequencies used, so adding an extra bit per cycle; phase-shifting keying technology (1200 bps)
• 1992 – Add error checking routines to existing technology to combat the echo problems, so modems get faster (9600bps).
• 1995 –  Added  Trellis modulation –  where the data in need of processing is placed into 2D structures in a logical manner, allowing mass error-checking – speeds go to 28K then to 56K…

Then cable arrives and changes the environment.  We can see steady evolution of speed to meet the demands of the users (selection) but each new advance builds upon (retains a legacy) and not replaces the old – until the environment changes; some technologies adapt, some go extinct.

Indeed this is a co-evolutionary process as the creation of more content heavy websites pushes demand for faster connections, so content creators make use of the faster connections to push more content and so on.

All of the examples discussed so far relate to the phenotype – what we can see.  It was all Darwin had – but it is not all we have today.  Biologists can delve into the DNA – the genotype to discover much more about the interrelationships between species; phylogenetics.

This image below is an image of the phylogenetic tree of fungus:

In my research, I am looking at software – specifically p2p clients.  This gives a distinct advantage over other approaches to technology because much of the development is open-source, so we can peer inside the technology and see how it was built.  We can start to look at it’s genotype.

Source-code is not an exact match to DNA – has similarities and differences, but the similarities are vital because they allow us to do what biologists have gone and track generational change.  So I have been looking at building proto-phylogenetics tree of p2p families.

Key ideas:

• Based on source-code relationships (akin to the genes in biology).
• Not based on the idea (the meme, though that is important too).
• Shows change over time. – each generation (release) of the software is the offspring.

So I looked to develop a method to find the family trees in p2p clients.  In a spreadsheet, I create an entry for each type of torrent client software – the full method and more information is here and a summary is below:

This graph (above) is the result of that method.  It shows releases of p2p clients over time…. (bigger image):

Next we add in the code/genetic links between the p2p clients (I.e. where one software project is based on the source code of a past project.)

Then by removing the background legends of the graph, we can pull out the famliy trees showing a basic (yes/no, not by degree) phylogenetic relationship.  The one below is for Azureus:

And BitTorrent:

And LibTorrent:

Now if we plot the changes of new releases (new offspring) over time from the last data-set into a new graph we also see something very interesting. Initially there is a small amount of activity.  Then  there is a growth of different clients and versions of clients in the middle.  The is a huge rise in the population (… many more individuals of each species are born than can possibly survive) and finally a drop-off of activity as the best p2p clients dominate the space:

So what does this mean?  This means that the lessons of biology can (sort of) be applied:

• Killing off individual sub-species, unless you get 100% of them (and the meme too!) far from stopping the spread of that species as a whole, may actually strengthen the species.  e.g see BitTorrent as a sub-species of the p2p (torrent) species and by removing that sub-species, provided at least one project survives, then the this variant passes the adaptations that mike it fitter on to the following versions.  It’s the same principle as to why antibiotics begin to fail.
• Variation is inevitable; the ease of development and branching creates more variation – which is why Open Source is so variable and thus it’s species are very strong (over all) against attack.
• Prediction is almost impossible.  What will be next?  Given the number of ideas, platforms and technologies – who knows?  Evolution is blind.  It only adapts to the now.  Users act as a component of ‘selection’; the inconsistencies in choices acts as – for all intents and purposes – natural selection. Clay Shirky had a great Twitter message that puts that point into perspective nicely:  “Why I ignore all “5 year plans”: 5 years ago, YouTube and Twitter didn’t exist, and Facebook was only for college kids” – Twitter itself came as a result of a flexible approach to technology from a company that was doing the same sort of thing as everybody else and realising it needed change.
• Adaptive technologies will win the evolutionary arms race.  The technologies that produce new generations and variation will eventually win.

Thanks! You can follow my research via Twitter (@TomasRawlings).