It might not make a great b-movie from the late 50’s, but the rise of the algorithms over the last decade has been just as thrilling, spectacular, and yes, sometimes even scary.

I was first turned on to the rise of algorithms by the founder of the gaming company Area/Code , Ken Slavin, and his fascinating 2011 talk on the subject at TED.  I was quickly draw to one Slavin’s illustration of the new power of algorithms in the world of finance.  Algorithms now control more than 70% of US financial transactions meaning that the majority of decisions regarding the buying and selling of assets are now done by machines. I initially took, indeed I still take, the rise of algorithms in finance to be a threat to democracy. It took me much longer to appreciate Slavin’s deeper point that algorithms have become so powerful that they represent a new third player on the stage of human experience: Nature-Humanity-Algorithms. First to finance.

The global financial system has been built around the electronic net we have thrown over the world. Assets are traded at the speed of light. The system rewards those best equipped to navigate this system granting stupendous profits to those with the largest processing capacity and the fastest speeds. Processing capacity means access to incredibly powerful supercomputers, but the question of speed is perhaps more interesting.

Slavin points out how the desire to shave off a few milliseconds of trading time has led to the hollowing out of whole skyscrapers in Manhattan. We tend to think of the Internet as something that is “everywhere” but it actually has a location of sorts in the form of its 13 core root servers through which all of its traffic flows. The desire to get close to route servers and therefore be able to move faster has led not only to these internally re-configured skyscrapers, but the transformation of the landscape itself.

By far the best example of the needs of algorithms shaping the world is the 825 mile fiber optic trench dug from Chicago to New York by the company Spread Networks. Laying the tunnel for this cable was done by cutting through my formidable native Alleghenies rather than following, as regular communications networks do, the old railway lines.

Slavin doesn’t point this out, but the 13 milliseconds of trading advantage those using this cable is only partially explained by its direct route between Chicago and New York. The cable is also “dark fiber” meaning it does not need to compete with other messages zipping through it. It’s an exclusive line- the private jet of the web. Some alien archaeologist who stumbled across this cable would be able to read the economic inequality endemic to early 21st century life. The Egyptians had pyramids, we have a tube of glass hidden under one of the oldest mountain ranges on earth.

Perhaps the best writer on the intersection of digital technology and finance is the Wall Street Journal’s  Scott Peterson with books like his Quants, and even more so his Dark Pools: The Rise of the Machine Traders and the Rigging of the U.S. Stock Market. In Dark Pools Peterson takes us right into the heart of new algorithm based markets where a type of evolutionary struggle is raging that few of us are even aware of. There are algorithms that exist as a type of “predator” using their speed to out maneuver slow moving “herbivores” such as the mutual funds and pension funds in which the majority of us little-guys, if we have any investments at all, have our money parked. Predators, because they can make trades milliseconds faster than these “slow” funds can see a change in market position- say selling a huge chunk of stock- and then pounce taking an advantageous position relative to the sale leaving the slow mover with much less than would have been gained or much more than would have been lost had these lightning fast piranhas not been able to strike.

To protect themselves the slow moving funds have not only established things like “decoy” algorithms to throw the predators off their trail, but have shifted much of their trading into non-public markets the “dark-pools” of Peterson’s title. Yet, even these pools have become infected with predator algos. No environment is safe- the evolutionary struggle goes on.

However this ends, and it might end very badly, finance is not the only place where we have seen the rise of the algorithms. The books recommended for you by Amazon or the movies Netflix informs you might lead to a good movie night are all based on sophisticated algorithms about who you are. The same kinds of algorithms that try to “understand” you are used by online dating services or even your interaction with the person on the other end of the line at customer service.

Christopher Steiner in his Automate This  points out that little ditty at the beginning of every customer service call “this call may be monitored…” is used not so much as we might be prone to think it is- a way to gauge the performance of the person who is supposed to help you with your problem as it is to add you to a database of personality types.  Your call can be guided to someone skilled in whatever personality type you have. Want no nonsense answers? No problem! Want a shoulder to cry on? Ditto!

The uber-dream of the big technology companies is to have their algorithms understand every element of our lives and “help” us to make decisions accordingly. Whether or not help should actually be in quotes is something for us as individuals and even more so as a society to decide with the main questions being how much of our privacy are we willing to give up in order to have smooth financial transactions, and is this kind of guidance a help or a hindrance to the self-actualization we all prize?

The company closest to achieving this algorithmic mastery over our lives is Google as Steven Kovach points out in a recent article with the somewhat over the top title Google’s plan to take over the world. Most of us might think of Google as a mere search company that offers a lot of cool compliments such as Google Earth. But, as its founders have repeatedly said, the ultimate goal of the company is to achieve true artificial intelligence, a global brain covering the earth.

 Don’t think the state, which Nietzsche so brilliantly called “the coldest of all cold monsters”  hasn’t caught on to the new power and potential of algorithms. Just as with Wall Street firms and tech companies the state has seized on the capabilities of advances in artificial intelligence and computing power which allow the scanning of enormous databases. Recent revelations regarding the actions of the NSA should have come as no surprise. Not conspiracy theorists, but reputable journalists such as the Washington Post’s Dana Priest  had already informed us that the US government was sweeping up huge amounts of data about people all over the world, including American citizens, under a program with the Orwellian name of The Guardian.  Reporting by James Bamford of Wired in March of last year had already informed us that:

In the process—and for the first time since Watergate and the other scandals of the Nixon administration—the NSA has turned its surveillance apparatus on the US and its citizens. It has established listening posts throughout the nation to collect and sift through billions of email messages and phone calls, whether they originate within the country or overseas. It has created a supercomputer of almost unimaginable speed to look for patterns and unscramble codes. Finally, the agency has begun building a place to store all the trillions of words and thoughts and whispers captured in its electronic net.

The NSA scandals have the potential of shifting the ground under US Internet companies, especially companies such as Google whose business model and philosophy are built around the idea of an open Internet. Countries have even more reason now to be energetic in pursuing “Internet sovereignty”, the idea that each county should have the right and power to decide how the Internet is used within its borders.

In many cases, such as in Europe, this might serve to protect citizens against the prying eyes of the US security state, but we should not be waving the flag of digitopia quite yet. There are likely to be many more instances of the state using “Internet sovereignty” not to protect its people from US snoops, but authoritarian regimes from the democratizing influences of the outside world. Algorithms and the ecosystem of the Internet in which most of them exists might be moving from being the vector of a new global era of human civilization to being just another set of tools in the arsenal of state power. Indeed, the use of algorithms as weapons and the Internet as a means of delivery is already well under way.

At this early date it’s impossible to know whether the revolution in algorithms will ultimately be for the benefit of tyranny or freedom. As of right now I’d unfortunately have to vote for the tyrants. The increase in the ability to gather and find information in huge pools of data has, as is well known, given authoritarian regimes such as China the ability to spy on its netizens that would make the more primitive totalitarians of the 20th century salivate. Authoritarians have even leveraged the capacity of commercial firms to increase their own power, a fact that goes unnoticed when people discuss the anti- authoritarian “Twitter Revolutions” and the like.

Such was the case in Tunisia during its revolution in 2011 where the state was able to leverage the power of a commercial company- FaceBook- to spy on its citizens. Of course, resistance is fought through the Internet as well. As Parmy Olson points out in her We Are Anonymous it was not the actions of the US government but one of the most politically motivated of the members of the hacktivist groups Anonymous and LulzSec, a man with the moniker “Sabu” who later turned FBI informant that was able to launch a pushback of this authoritarian takeover of the Internet. Evidence if there ever was any of that hacktivism even when using Distributed Denial of Service Attacks (DDOS) can be a legitimate form of political speech

Yet, unlike in the movies, even the rebels in this story aren’t fully human. Anonymous’ most potent weapon DDOS attacks rely on algorithmic bots to infect or inhabit host  computers and then strike at some set moment causing a system to crash  due to surges in traffic. Still, it isn’t this hacktivism of groups like Anonymous and LulzSec that should worry any of us, but the weaponization of the Internet by states, corporations and criminals.

Perhaps the first well known example of a weaponized algorithms was the Stuxnet Worm deployed by the US, Israel, or both, against the Iranian nuclear program. This was a very smart computer worm that could find and disable valuable pieces of Iran’s nuclear infrastructure leaving one to wonder whether the algo wars on Wall Street are just a foretaste of a much bigger and more dangerous evolutionary struggle.

Hacktivist groups like Anonymous or LulzSec have made DDOS attacks famous. What I did not know, until I read Parmy Olson, is that companies are using botnets to attack other companies as when Bollywood used the company AiPlex to attack the well known copyright violators such as Pirate Bay by using DDOS attacks. What this in all likelihood means is that AiPlex unknown to their owners infiltrated perhaps millions of computers (maybe your computer) to take down companies whose pirated materials you might never have viewed. Indeed, it seems the majority of DDOS attacks are little but a-political thuggery- mobsters blackmailing gambling houses with takedowns on large betting days and that sort of Sopranosesque type of thing.

Indeed, the “black-hats” of criminal groups are embracing the algorithmic revolution with abandon. A lot of this is just annoying: it’s algorithms that keep sending you all those advertisements about penis enlargement, or unclaimed lottery winnings, but it doesn’t stop there. One of the more disturbing things I took away from Mark Bowden’s Worm the First Digital War   is that criminals who don’t know the first thing about programming can buy “kits”, crime algorithms they can customize to, say, find and steal your credit card information by hacking into your accounts. The criminal behind this need only press a few buttons and whola! he’s got himself his very own cyber-burglar.

 The most advanced of these criminal algorithms- though it might be a weapon of some state or terrorist group, we just don’t know- is the Conficker Worm, the subject of Bowden’s book which was able to not only infects millions of computers by exploiting a whole in Windows- can you believe it?!- but has created the mother of all botnets, an algorithm capable of taking down large parts of the Internet if it chose, but for whatever reason just sits there without doing a damned thing.

As for algorithms and less kinetic forms of conflict, the Obama Campaign of 2012 combined the same mix of the capability to sort huge data sets combined with the ability to sort individuals based on psychological/social profiles that we see being used by tech companies and customer service firms. Just like the telemarketers or CSRs the Obama campaign was able to tailor their approach  to the individual on the other end of their canvasing – amplifying their persuasive power. That such mobilizing prowess has not also led to an actual capacity to govern is another matter.

All this is dark, depressing stuff, I know. So, I should end with a little light. As Steiner points out in his Automate This, our new found power to play with huge data sets, and,  in what sounds like an oxymoron, customize automation, promises a whole host of amazing benefits. One of these might be our ability to have a 24/7 personal AI “physician” that monitors our health and drastically reduces medical costs. A real bone for treating undeserved patients whether in rural Appalachia or the develping world.

Steiner is also optimistic when it comes to artists. Advanced algorithms now allow, and they’ll just get better, buyers  to link with sellers in a way that has never been possible before. A movie company might be searching for a particular piece of music for their film. Now, through a service like Music-Xray  the otherwise invisible musician can be found.

Here I have to put my pessimist cap back on for just a minute, for the idea that algorithms can help artists be viable, as of this writing, is just that a hope. Sadly, there is little evidence for it in reality. This is a point hit home by the recent Atlantic Online article: The Reality of the Music Business Today: 1 Million Plays = $16.89. The algorithm used by the Internet music service, Pandora, may have helped a million people find musician David Lowery and his song “Low”, but its business model seems incapable of providing even successful musicians with meaningful income. The point that the economic model we have built around the “guy with the biggest computer” has been a bust for artists of all sorts is most strongly driven home by the virtual reality pioneer and musician Jaron Lanier. Let’s hope Lanier is ultimately wrong and algorithms eventually provide a way of linking artists and their patrons, but we are far, far from there yet. At the very least they should provide artists and writers with powerful tools to create their works.

Their are more grounds for optimism. The advance of algorithms is one of the few lit paths out of our current economic malaise. Their rise appears to signal that the deceleration in innovation which emerged because of the gap between the flood of information we could gather, the new discoveries we were making, and our ability to model those discoveries coherently, may be coming to an end almost as soon as it was identified. Advanced algorithms should allow us to make potent and amazing new models of the natural world. In the long run they may allow us to radically expand the artistic, philosophical and religious horizons of intelligence creating visions of the world of which we can today barely dream.

On a more practical and immediate level, advanced algorithms that can handle huge moving pieces of information seem perfect for dealing with something like responding to a natural disaster or managing the day to day flows of a major city such as routing traffic or managing services- something IBM is pioneering with its Smart Cities Projects in New York City and Rio De Janeiro.

What algorithms are not good at, at least so far, and we can see this in everything from the Obama campaign in light of its political aftermath, to the war on terrorism, to the 300,000 person protests in Rio this past week despite how “smart” the city, is expanding their horizon beyond the immediate present to give us solutions for the long term political, economic and social challenges we confront, instead merely acting as a globe- sized amplifier of such grievances which can bring down governments but not creating a lasting political order.  To truly solve our problems we still need the mother of all bots, collective human intelligence. I am still old fashioned enough to call it democracy.