AI Panic!

About 9 years ago, on June 10th 1999, the first humans fell victim to a computer control system failure. Due to a chain of unfortunate circumstances, a the pressure on a 16-inch steel pipe line increased uncontrollably and caused it to rupture, flooding two small creeks near Bellingham in the process. The petrol ignited and killed two 10-year-old boys and an 18-year-old adolescent. Although it was later revealed that the accident was in part caused by human failure, it is still considered as the first cyber-event that killed humans.

Now, you might think that things have changed since then, but you’d be wrong. Joe Weiss from Applied Control Solutions says:

Until eight years ago, my whole life was making control systems usable and efficient, and, by the way, very vulnerable. It is exactly what you will find today in many, many industrial applications. This isn’t just 1999. No, this is June 2008.

And it is true. The computer systems that control complex technical facilities, like the power grid, traffic control infrastructure, or pipeline systems have historically only been optimized with respect to usability, efficiency and effectivity. Security has almost never been a concern in the design of these systems. Many of them have been developed before the rise of the internet, so that attacks had to be carried out physically. But eventually some systems responsible for the control of important infrastructure have been connected to a network, often in an effort to improve control or just because of company policies, without any evaluation of possible risks.

There are more examples. The 787 Dreamliner from Boeing, which is scheduled to come out later this year, may be vulnerable to hacker attacks. This is because the flight control network is separated from the passenger network only by software firewalls, and thus might be attacked from inside the plane (or even from the outside, with the help of a trojan horse maybe).

When military control systems fail, it gets nasty quite quickly. Last year, a robot anti-aircraft cannon killed 9 and wounded 14 soldiers when it went berserk during a training of the South African National Defence Force. However, it is not yet clear if the accident was caused by a mechanical malfunction or software failure.

And it is not only the security of control systems that poses a risk, the control systems themselves are feared by researchers to remove human control from key decision-making processes. Tom Rodden, Professor of Interactive Systems at the University of Nottingham concluded at the Human-Computer Interaction conference that this loss of control might not directly be a risk for our lives, but surrenders “basic human values and concepts such as personal space, society, identity, independence, perception, intelligence and privacy.”

This lack of security must become a higher priority to control system designers, especially as there is an unavoidable trend towards more autonomous, intelligent systems that can cope with the size and complexity of modern infrastructure. We have to take care that we do not hand over all control of critical systems and uncritical but socially important concepts to machines. Even if a worst case scenario - such as malicious AI breaking loose - does not happen, programs are likely to be susceptible to bugs and misinterpreted signals, and therefore have to be very carefully designed. As the examples in the past have shown, these issues are a real threat to human beings and the lack of security against tampering from the outside increases the AI Panic level by +1%.

Imagine, someday, we’ll lose the control over our computers to some sentient being, unable to stop it or to unplug the millions of home pcs, office workstations or lab mainframes affected.

Well, there is no need to imagine, this very thing has been happening for a long time now, in the form of Zombie Computers. These machines have been compromised by hackers, viruses or trojans, and are remote controllable. Often they are grouped in a network of zombie computers, called a botnet.

The RSA 2008 conference in San Francisco earlier this month had a panel discussion on this topic, and the tone was “a mix of resignation, indignation and post-9/11 rhetoric”, according to Threat Level. According to the panelists, botnets are the biggest threat in the internet today and a danger to national security.

However, little is done to combat the use of zombie computers. Although some botnet operators are caught once in a while, the spam and attacks they create does not seem to decrease. I agree with Ira Winkler, a security consultant who said at the conference:

“The problem is no one is doing anything. Guess what? If your system has a bot on it, you don’t get on the internet. [...] We need to hold people responsible when they present an imminent threat to other people.”

However, ISPs are reluctant to cut off offenders, as they often have no clue what’s going on and respond with confusion and angry phone calls, which drive up the customer support costs.

The inability to quickly and efficiently stop these bot nets could prove very helpful for a malevolent AI. Once released on the internet (or on its successor, maybe The Grid), there is virtually no way to stop it again. The infrastructure to spread uncontrollably is already given by the botnets. Unless there is some fundamental change on the network level and software level (hello Microsoft!), the only way to stop such an outbreak would be the complete shut-down of the net. Of course, this would cause quite a disturbance to the world economy, as most financial transactions and international trading are entirely done online. Most nations will be hesitant to risk that, just because some strange program has been set free from a university lab somewhere.

Therefore I think the current infrastructure of the web eases the spread of an AI (Panic Level: +3%), once it discovers the weaknesses of connected hosts and “understands” how botnets work. But that’s not yet an immediate threat; after all we have to develop the artificial general intelligence first, which, mind you, is a little tricky.

It sounds like the sermon Noel Sharkey preached last month about the ethics of autonomous systems has been really necessary. The time-frame we are talking about here is smaller than one might think - I suspect just 10-15 more years and we will see autonomously shooting robots. If there is a war at that time and morality debates fail, that is.

A reporter from Popular Mechanics visited an U.S. Army test area in Texas recently and wrote about the demonstration of the “MULE”, a six-wheeled utility vehicle, which is currently remote controlled by an Xbox 360 controller, but the developers at Lockheed Martin plan to include autonomous movement by the scheduled release date 2014. Already 1700 of these Humvee sized beasts have been ordered by the U.S. Army, and while they say that the use of the mounted weapons (4 antitank missiles and a machine gun) in this generation of robots will always have to be initiated by a human controller, I suspect that this will be “optional” in future versions. Maybe in the form of a bAutoFire = false; that can easily be overridden in desperate situations after shouting a key phrase like It’s coming right for us!

The article questions the readiness of the robots for battle, and comes to the conclusion that there’s still a long way to go and the robots are still clumsy and slow. However, the hardware is completely ready for battle (and probably has been for a while). It is just the software that is still missing. Just some little intelligent pathfinding, friend/foe distinction, an aimbot and a wallhack and we’re all set!

Well, of course it is not that easy. Indeed, the artificial intelligence is by far the most difficult problem these robots face, and as mentioned, it’ll take another couple of years until we see reliable algorithms that prevent friendly fire (My guess would be that they come up with a hardware based solution first, an active transmitter or reflector that indicates a friend. However, that could become problematic when soldiers are captured.)

Eventually, the software side will be solved, and only the ethical issue will remain. War ethics are a difficult terrain and probably won’t count for a lot when an officer sees his soldiers at risk and decides to push the button. The huge budget of the U.S. Army lures researchers and universities like Carnegie Mellon to put their brainpower to use on modern AI, AI that will be used to control weapons to kill people.

Now I’ve rambled along way too much again, time to finish up. I’ll increase the AI Panic Level by +1% (not more because I’ve already credited the general risk of autonomous army robots in the post about Noel Sharkey).

 Addition (March 22nd, 2008): There are also flying prototypes, like that bat spy plane shown below. The U.S. Army awarded the University of Michigan College of Engineering a five-year, $10-million grant to construct and build it. While it sure is more difficult to build a flying object than an earthbound robot, the challenges lie more in the physical realization than in a better AI.

Unmanned flying drones are in use by the military already and have been used in Iraq to fire missiles and collect terrain information.

Thanks Igor Gabrielan and Erk Subasi for letting me know!
If you see something in the net that would make a good AI Panic article, feel free to contact me!

What if the people you talk to, play with and interact with in Second Life weren’t human at all? Granted, none of them is literally human anyway, but almost all of them are actually controlled by someone sitting in front of a computer. Not all are.

You might run into a little, 4 year old child that appears to be reasonably smart and is able to reason about its surroundings. This child is controlled by a program. A program that is able to reason about its own beliefs to draw conclusions in a manner that matches human children its age.

The program is the result of research conducted by a group around Selmer Bringsjord at the Rensselaer Polytechnic Institute. According to ScienceDaily, Bringsjord says:

Truly convincing autonomous synthetic characters must possess memories; believe things, want things, remember things. Such characters can only be engineered by coupling logic-based artificial intelligence and computational cognitive modeling techniques with the processing power of a supercomputer. [...] The logico-mathematical theory will include rigorous, declarative definitions of all of the concepts central to a theory of the mind, including lying, betrayal, and even evil.

Using Second Life as a platform for this makes sense. Reducing the communication bandwidth is a common trick to lower the barrier of believability. Even the good ol’ Alan Turing limited the interaction for his famous Turing test to test if a computer can demonstrate intelligence to a text chat. Second Life is reasonably close to that, avatars can easily be controled by keyboard commands.

Building agents based on the theory of mind (which describes the ability to attribute mental states to oneself and others and to understand that others have beliefs, desires and intentions that are different from one’s own) is a step towards an artificial general intelligence, and thus towards a real threat (or a great opportunity, depends on who you ask). And indeed, this project was presented a couple of days ago at the Artificial General Intelligence conference held at the University of Memphis.

I believe there is a high risk connected to the creation of such an intelligence, it is easy to think of hypothetical scenarios in which a reasoning system concludes that humans are a risk and need to be wiped out. Although testing such an agent in a virtual world is a good idea to realise what it is capable of and evaluating the actual risk of it, once the tools are there the step into the real life will be taken quickly. And sure enough, Selmer Bringsjord has plans to apply the technlogy to education and homeland defense. Good thing the system has no sense of lying, betrayal and isn’t evil…

The researchers want to build an interface with a certain resemblance to the holodecks in Star Trek for direct interaction of agents with human beings. While this definitely sounds like science fiction, it has to be taken seriously. They have the support of strong industrial partners and want to use the worlds most powerful university-based supercomputing system at the CCNI.

The research is still in an early stage and the Second Life results are far away from being strong AI, but they might well bring us closer to an artificial general intelligence, thus raising the AI Panic Level by +1%. I don’t know whether the system they are working on will achieve a level of strong AI (most likely not), and even if it does, we don’t know if it will be good or bad. I’ll finish with a quote from the group’s research homepage, which interestingly enough seems to answer that thought:

An advanced synthetic character, however, can literally be evil, because it has the requisite desires, beliefs, and cognitive powers.

Noel Sharkey, a professor at the University of Sheffield, warned conference attendees today about the danger of an arms race in robotic warfare.

He gave a keynote speech at the “Ethics of Autonomous Military Systems” event hold at the Royal United Services Institute (RUSI) for Defence and Security Studies, warning the audience about terroristic aspects of war robots. He argues that terrorists will use these robots for their purposes:

“Once the new weapons are out there, they will be fairly easy to copy. How long is it go­ing to be be­fore the ter­ror­ists get in on the ac­t? [...] With the cur­rent prices of ro­bot con­struc­tion fall­ing dra­mat­ic­ally and the avail­abil­ity of read­y-made com­po­nents for the am­a­teur mar­ket, it would­n’t re­quire a lot of skill to make au­ton­o­mous ro­bot weapons.” Sharkey said a small GPS-guided drone with au­to­pi­lot can be made for about $500.

Interestingly, he accounts the dumbness of these robots for part of the potential danger, because the robots might not be able to distinguish between friend or foe.

Military technology expert James Canton compared the development of robot war technologies with Moore’s Law, but “on steroids.” It is only a matter of time before the human factor is completely removed from the control of war robots. For now, the trigger has still to be pulled manually, as the control logic is not yet good enough and ethical questions still have to be answered.

However, discussing the morality and ethicality of warfare is hypocritical anyway, and has rarely stopped the use and development of military equipment (with the notable exception of biological and chemical weapons). Now as semi-automatic robots are used in real-”life” already, I doubt that a noticeable opposition will be formed, especially as the effects and danger of these weapons are not directly visible — on the contrary, the accuracy will improve and friendly fire is likely to reduce. I think only a mayor accident will be enough to change the perception of these robots, but then it is already too late. As soon as the technology is out there, it is available to be used by terrorist organisations.

While terrorist attacks are not directly related to AI, the development of autonomous robots is. Many research institutes and universities are striving towards automated image recognition, which is the major hurdle for the development of fully autonomous war robots, as the Geneva Convention requires the reliable distinction between combatants and civilians. Warnings from Experts like Noel Sharkey indicate the reality of the danger of autonomous war robots, which is why the AI Panic Level will be increased by +5%.

[Sources: AFP, World Science, The Register]

Ray Kurzweil gave a keynote speech at this years GDC in which he adressed the future of gaming (a bit) and the future of mankind (a lot). Most of the readers of this blog are probably familiar with Ray Kurzweil and his theories. In short, he proposed the law of accelerating returns which will lead to a technological singularity in about 20-30 years. He extends Moore’s Law to the future and uses it (among other indicators) to predict future values for chip densities, prices, computing power, etc. The exponential acceleration of these values will lead to massive and fundamental changes in the way we live and work, culminating in self-replicating and -improving machines, which triggers an explosion in technological advancement, as the human as innovating factor is eliminated.

As far as I can tell, the keynote speech last Thursday did not contain any results that have not been discussed in one of his previous talks about that matter, Kurzweil just added some game related stuff at the beginning, saying how “game” was an unfortunate name, because it stressed the disconnection from reality, but games are increasingly interwoven with reality (just like talking to someone over the telephone is not virtual). More info on the keynote can be found at 1up and the Escapist.

The law of accelerating returns and the technological singularity pretty much state that a strong AI will be created in the very near future. The singularity, if it so happens, will most likely produce an intelligence superiour to our own. While Ray Kurzweil sees a positive future here, talking about eternal life and friendly AI, this vision seems unlikely to me. Especially considering the inherent malevolence of generic AI algorithms, the probability of a hostile AI is high, I rise the AI Panic Level by +10%. This rise is not an effect of the keynote speech from Ray Kurzweil, but more a result of the discussion of the singularity and accelerating return theories.

The theories are compelling. The statistical foundation Kurzweil provides is reasonable and I can’t see many reasons why Kurzweils predictions should be wrong (a nuclear war would be one, but we surely do not want that!).

Alas, we’re up for some interesting times in the next 25 years, lets try to make the best of it! By which I mean, proactively promote the need for AI safety regulations (for example in the form of technological frameworks or social infrastructures)!

Scientists from the Ural University of Physics and Technology have developed components required for artificial consciousness according to the “consciousness electromagnetic information field” theory of Johnjoe McFadden. This theory states that the electromagnetic field induced by the neurons in the brain carries human consciousness.

Russia IC reports in somewhat broken English that the Russian think tank “created a model of neural network on neurons with additional channels for information exchange via electromagnetic field.” Apparently the artificial neurons they created are very similar to the EM neuron concept from McFadden.

It is not really clear from the article exactly how powerful or useful their hardware is, however, for the Russian scientists it seemed good enough to patent it. So far, I haven’t seen a single convincing use of artificial neural networks apart from academic toy problems. There are implementations here and there that use them, but almost everywhere it is easier and faster to just use other, more predictable and understandable techniques. Universities hype them in AI lectures (which is ok, they are relatively easy and have a nice theory behind them), and I believe that’s why many try to apply them everywhere they can (which is also ok, but only as long as they have good reasons to do so — which they usually don’t).

Therefore I doubt they will achieve their goals of an artificial consciousness, especially as they themselves don’t know how or if spontaneous generation of consciousness in this kind of networks works at all. It is not impossible though, therefore I’ll rise the AI Panic Level by +0.1%. After all, the Flying Spaghetti Monster must have had some motivation to endow us with a natural neural network …

The DARPA has announced that it will be using a Generalized Integrated Learning Architecture (GILA) system from Lockheed Martin to manage crowded airspace. It is intended to help the Air Force to cope with increased air traffic, especially as unmanned aerial vehicles and other airborne weapons get more common.

In effect, the DARPA is planning to hand over the control of missiles to an AI system which is able to learn and reason on its own. Brilliant! According to DARPA, the software will combine reasoning systems, general knowledge and by asking what-if questions.

“The integrated learner also will have explicit learning goals, keep track of what it does not know, what it needs to know, as well as track and reason about its uncertainties. The software will attempt to figure things out, as well as tolerate errors and missing information by using whatever information or reasoning is available. Integrated learning software must be able to manipulate many different forms of information and even trade off different types of information and reasoning, as well as interact with humans to fill in information gaps.”

Let’s just hope it does not learn from the wrong examples, or find out that it can work much more efficiently when it completely shuts off human influence from its controls. Many researchers believe that such reasoning systems are not unlikely to come to wrong solutions that result for example because of poor problem domain description (e.g., the goal might be to minimise the probability of a crash at a given future date. A solution is to crash all planes now, then there are no planes in the future, i.e., the probability of a crash will be zero …). Also, a learner that learns from positive and negative examples is prone to misclassifications. Therefore I’ll rise the AI Panic Level by +1%.

I believe that it might be hard to spot errors in the rule base and inconsistencies in the examples used to train the software. A system able to predict outcomes based on examples has to have some kind of bias, a restriction to possible world states. This bias must be chosen carefully to avoid empty solution sets (too strict restrictions) or full sets (too lax restrictions, the system is unable to make useful predictions).

As long as this system is used in training and double-checking the flight operators decisions only, there is no direct danger. But if DARPA decides to go live with this system (which might never happen though), it is of course critical that no errors occur. This depends crucially on the proper set-up of the background knowledge and underlying rule bases. Lets hope they get it right and make no precipitous moves to replace those error-prone humans!

The Gulf Cooperation Council might have to rely on artificial intelligence technologies to fight unemployment, GulfNews.com reports. Joachim Diedrich, a professor of artificial intelligence at the Uni of Queensland, Australia, embraces this dependency:

“For decades now, there has been a misconception that artificial intelligence will replace human resource. Rather, it is a job creator. [...] I see this as a positive opportunity. We should be optimistic about the availability of new jobs through the development of artificial intelligence.”

Probably we should indeed be optimistic, as there might not be a way around a tighter integration of AI into our daily life and products anyway. This integration has long started, it is being used in predicting weather and the stock market, so there is no way (and use) of denying its importance to the job market.

Although Diedrich tries to allay security doubts by saying that he does not think that “the industry will come to depend on artificial intelligence,” he lowers the credibility of his own statement with his next: “The use of artificial intelligence is not public knowledge, we don’t know how or where it is being used in the region.”

This far-reaching integration of black-box AI into important economical factors like the stock exchange and a huge number of jobs creates a dangerous dependency with regards to the worst case scenario of a hostile AI (it increases the AI Panic Level by +2%).

While current AI methods, used for example to predict the weather or stock values, are quite simple and pose no threat whatsoever, it will get more and more difficult to say that of future techniques. The accelerating technological advancement will enable very complex systems that incorporate extensive background knowledge from news articles, common sense databases and historical data to optimise the accuracy of predictions. This accurate world model might enable AIs to reason about everyday life and is a requirement for general artificial intelligence.

Thus a high job dependency on AI in this area will feed into the advancement and research of this area and brings us closer to a possibly dangerous strong AI.

A new study by ABI Research shows that by 2015 the market for personal robotics components might reach $12 billion. They expect task robotics (that ease routine chores like cleaning, driving, etc) and entertainment robotics to have roughly equal parts in this revenues.

The computerworld article goes on to speculate that a part of the entertainment sector will be made up by human-like robots that can act as companions (see also my article on seducing robots for more on this).

By 2015, ABI Research expects more than 21 million robots shipped yearly, and predicts very high demand in sensors, actuators, servos and manipulators.

This study indicates that elaborate robots will penetrate the market very soon, which is only the next step after “intelligent” microcontrollers, that can nowadays be found everywhere, in cars, toys and even rice cookers. This increased dependency on machines with a high level of computerisation and connectivity together with a lacking sense for the possible threats of advanced AI is dangerous, the AI Panic Level increases by +3%. In the worst case scenario (which we have to take care of) robots with a network connection could be remotely controlled or reprogrammed (for example through a tampered firmware update) to follow the commands of an AI that went berserk.