AI Panic!

London-based charity Landmine Action wants autonomous robots capable of killing people banned under the same kind of treaty that has outlawed landmines in over 150 countries. According to the New Scientist it is the first time a high profile non-governmental organisation has campaigned against such a technology. This campaign follows the reasoning of Noel Sharkey, who condemned these automation plans earlier this year.

As I’ve written before, the robots in use by the military nowadays (and the next years) are almost fully automatic, but so far the trigger has still to be pulled by a human soldier. However, it is only a question of time until the software is strong enough so that this decision will be made entirely by the machine. And once the software is in place, there will be no ethical opposition - at least in the US Department of Defence, who wants them in future to work without supervision.

A reaction to this news article comes from Wired, where the idea of danger through war robots is dismissed:

But to argue as if this is in the here or now, or even in the next decade, is just plain silly. The Pentagon has not only never advocated taking the man-out-the-loop of targeting decisions for drones or robots, its current policies and procedures would prohibit such a move (some might argue that international law already prohibits autonomous armed drones). [...] Unless and until those policies are drastically altered, it’s safe to say we are safe from renegade Terminators.

To completely ignore the threat by robots with weapons and justifying this ignorance by saying that these robots are still science fiction and it will take decades until these robots appear seems a bit strange to me, especially as Wired itself has reported about existing armed robots before. It is exactly this ignorance and belief that everything is so far away that allows organisations like the military to push these developments without any opposition.

Furthermore, I think it is a little blue-eyed to think that the control of weapons mounted on otherwise fully automated robots will remain in human hands just because the Pentagon does not admit of having plans that say otherwise. By arguing that international law might already prohibit autonomous armed drones and at the same time clearly seeing that these very drones are being used by the U.S. Army in Iraq right now, Wired maneuvers itself on very thin ice. I don’t think it can support the arguments that promises, laws and policies are sufficient enough to protect us from a - what Wired deems nonexistant - danger. And heck, what is so bad about protesting against future dangers as opposed to only trying to fight seeing the effects of existing weapons like landmines in hindsight?

I think it is about time that more voices are raised against automating war machinery, and the Landmine Action has taken a step in the right direction. If it will be heard by the military - and the U.S. Army is without doubt on the forefront of research towards this automation - is a completely different matter. So far, I think the chances are still slim, to specify I say it lowers the AI Panic Level by -0.1%.

It sounds like the sermon Noel Sharkey preached last month about the ethics of autonomous systems has been really necessary. The time-frame we are talking about here is smaller than one might think - I suspect just 10-15 more years and we will see autonomously shooting robots. If there is a war at that time and morality debates fail, that is.

A reporter from Popular Mechanics visited an U.S. Army test area in Texas recently and wrote about the demonstration of the “MULE”, a six-wheeled utility vehicle, which is currently remote controlled by an Xbox 360 controller, but the developers at Lockheed Martin plan to include autonomous movement by the scheduled release date 2014. Already 1700 of these Humvee sized beasts have been ordered by the U.S. Army, and while they say that the use of the mounted weapons (4 antitank missiles and a machine gun) in this generation of robots will always have to be initiated by a human controller, I suspect that this will be “optional” in future versions. Maybe in the form of a bAutoFire = false; that can easily be overridden in desperate situations after shouting a key phrase like It’s coming right for us!

The article questions the readiness of the robots for battle, and comes to the conclusion that there’s still a long way to go and the robots are still clumsy and slow. However, the hardware is completely ready for battle (and probably has been for a while). It is just the software that is still missing. Just some little intelligent pathfinding, friend/foe distinction, an aimbot and a wallhack and we’re all set!

Well, of course it is not that easy. Indeed, the artificial intelligence is by far the most difficult problem these robots face, and as mentioned, it’ll take another couple of years until we see reliable algorithms that prevent friendly fire (My guess would be that they come up with a hardware based solution first, an active transmitter or reflector that indicates a friend. However, that could become problematic when soldiers are captured.)

Eventually, the software side will be solved, and only the ethical issue will remain. War ethics are a difficult terrain and probably won’t count for a lot when an officer sees his soldiers at risk and decides to push the button. The huge budget of the U.S. Army lures researchers and universities like Carnegie Mellon to put their brainpower to use on modern AI, AI that will be used to control weapons to kill people.

Now I’ve rambled along way too much again, time to finish up. I’ll increase the AI Panic Level by +1% (not more because I’ve already credited the general risk of autonomous army robots in the post about Noel Sharkey).

 Addition (March 22nd, 2008): There are also flying prototypes, like that bat spy plane shown below. The U.S. Army awarded the University of Michigan College of Engineering a five-year, $10-million grant to construct and build it. While it sure is more difficult to build a flying object than an earthbound robot, the challenges lie more in the physical realization than in a better AI.

Unmanned flying drones are in use by the military already and have been used in Iraq to fire missiles and collect terrain information.

Thanks Igor Gabrielan and Erk Subasi for letting me know!
If you see something in the net that would make a good AI Panic article, feel free to contact me!

Noel Sharkey, a professor at the University of Sheffield, warned conference attendees today about the danger of an arms race in robotic warfare.

He gave a keynote speech at the “Ethics of Autonomous Military Systems” event hold at the Royal United Services Institute (RUSI) for Defence and Security Studies, warning the audience about terroristic aspects of war robots. He argues that terrorists will use these robots for their purposes:

“Once the new weapons are out there, they will be fairly easy to copy. How long is it go­ing to be be­fore the ter­ror­ists get in on the ac­t? [...] With the cur­rent prices of ro­bot con­struc­tion fall­ing dra­mat­ic­ally and the avail­abil­ity of read­y-made com­po­nents for the am­a­teur mar­ket, it would­n’t re­quire a lot of skill to make au­ton­o­mous ro­bot weapons.” Sharkey said a small GPS-guided drone with au­to­pi­lot can be made for about $500.

Interestingly, he accounts the dumbness of these robots for part of the potential danger, because the robots might not be able to distinguish between friend or foe.

Military technology expert James Canton compared the development of robot war technologies with Moore’s Law, but “on steroids.” It is only a matter of time before the human factor is completely removed from the control of war robots. For now, the trigger has still to be pulled manually, as the control logic is not yet good enough and ethical questions still have to be answered.

However, discussing the morality and ethicality of warfare is hypocritical anyway, and has rarely stopped the use and development of military equipment (with the notable exception of biological and chemical weapons). Now as semi-automatic robots are used in real-”life” already, I doubt that a noticeable opposition will be formed, especially as the effects and danger of these weapons are not directly visible — on the contrary, the accuracy will improve and friendly fire is likely to reduce. I think only a mayor accident will be enough to change the perception of these robots, but then it is already too late. As soon as the technology is out there, it is available to be used by terrorist organisations.

While terrorist attacks are not directly related to AI, the development of autonomous robots is. Many research institutes and universities are striving towards automated image recognition, which is the major hurdle for the development of fully autonomous war robots, as the Geneva Convention requires the reliable distinction between combatants and civilians. Warnings from Experts like Noel Sharkey indicate the reality of the danger of autonomous war robots, which is why the AI Panic Level will be increased by +5%.

[Sources: AFP, World Science, The Register]

The DARPA has announced that it will be using a Generalized Integrated Learning Architecture (GILA) system from Lockheed Martin to manage crowded airspace. It is intended to help the Air Force to cope with increased air traffic, especially as unmanned aerial vehicles and other airborne weapons get more common.

In effect, the DARPA is planning to hand over the control of missiles to an AI system which is able to learn and reason on its own. Brilliant! According to DARPA, the software will combine reasoning systems, general knowledge and by asking what-if questions.

“The integrated learner also will have explicit learning goals, keep track of what it does not know, what it needs to know, as well as track and reason about its uncertainties. The software will attempt to figure things out, as well as tolerate errors and missing information by using whatever information or reasoning is available. Integrated learning software must be able to manipulate many different forms of information and even trade off different types of information and reasoning, as well as interact with humans to fill in information gaps.”

Let’s just hope it does not learn from the wrong examples, or find out that it can work much more efficiently when it completely shuts off human influence from its controls. Many researchers believe that such reasoning systems are not unlikely to come to wrong solutions that result for example because of poor problem domain description (e.g., the goal might be to minimise the probability of a crash at a given future date. A solution is to crash all planes now, then there are no planes in the future, i.e., the probability of a crash will be zero …). Also, a learner that learns from positive and negative examples is prone to misclassifications. Therefore I’ll rise the AI Panic Level by +1%.

I believe that it might be hard to spot errors in the rule base and inconsistencies in the examples used to train the software. A system able to predict outcomes based on examples has to have some kind of bias, a restriction to possible world states. This bias must be chosen carefully to avoid empty solution sets (too strict restrictions) or full sets (too lax restrictions, the system is unable to make useful predictions).

As long as this system is used in training and double-checking the flight operators decisions only, there is no direct danger. But if DARPA decides to go live with this system (which might never happen though), it is of course critical that no errors occur. This depends crucially on the proper set-up of the background knowledge and underlying rule bases. Lets hope they get it right and make no precipitous moves to replace those error-prone humans!