This is the translation by John Holland of my post entitled Complexité et effet Skynet, originally published on June 16th 2012.

The world in which we live today increasingly resembles a science fiction film from the 1950s. Some of those films were excellent because they raised fundamental questions about how machines seize power. Put more simply, they explored how the invasion of the machines–an invasion we have actively fostered for several centuries–leads us to lose control of the world. The film 2001, A Space Odyssey (1968) culminates in a battle: the machine that is really in charge of the mission―and knows how to carry it out―is pitted against the human crew member whom the machine has been programmed to sacrifice. Incapable even of imagining that he is not in command, the human is led by his arrogance to fight back and eventually to escape, although the nature of his salvation turns out to be quite problematic.

Today’s financial markets are the prey of robots that fight their duels in computerized space. This is the result of high-frequency trading algorithms, or algos. Of course there are programmers behind these algos, who write the software and can evaluate its effects at day’s end, fine-tuning when necessary and adding improvements and innovations. Nevertheless, and this is just like neural networks and genetic algorithms (whose inner workings are impenetrable to human perception), certain machine-learning techniques produce highly autonomous behavior in algos as long as they remain in operation. The upshot of this is that the “Skynet effect,” as we have come to call it, is now in play. This is a reference to the omnipotent computer network featured in the Terminator series. In those films, human beings no longer have a role except in a broader context where major decisions are all really made by a confederation of computers.

We must ask ourselves today with the utmost seriousness whether we still have control of computers and robots (other than having the power to flip the “Off” switch, of course). If not, what do we need to do to get it back?

=============================
Since this post was written, its object was illustrated in the graphic novel La survie de l’espèce (Futuropolis 2012) by Gregory Maklès and myself.