Surely no harm could come from building a chess-playing robot, could it? In this paper we argue that such a robot will indeed be dangerous unless it is designed very carefully. Without special precautions, it will resist being turned off, will try to break into other machines and make copies of itself, and will try to acquire resources without regard for anyone else’s safety.

These are the first sentences of the paper “Basic AI Drives” from Stephen Omohundro. He argues that AI systems are inherently egoistic and careful design is necessary to prevent emergent behaviour from developing hostile self-defence mechanisms. If you read the paper — and I suggest you do, it’s only 11 pages and well written — you’ll see that an AI Panic Level rise of +10% is justified.

In his paper, Omohundro makes the following propositions:

• AIs will want to self-improve
• AIs will want to be rational
• AIs will try to preserve their utility functions
• AIs will try to prevent counterfeit utility (i.e., corruption)
• AIs will be self-protective
• AIs will want to acquire resources and use them efficiently

He concludes with a warning that we have to make appropriate changes soon and suggests to set up a “universal constitution” comprising the most essential rights of individuals.

In fact, while reading this paper, I was reminded of the Chernobyl disaster. Not because of the potential hazards, but more because of the reactor design: Back in the time of the disaster, many reactors possessed what is called an active safety system. Upon a failure or safety problem, this system had to be actively activated. That is similar to the design of an AI system, a lot of work has to be actively done to avoid mistakes and ill-behaviour.

Nowadays most reactors have passive nuclear safety features, i.e., they default to a safe system state once something goes wrong. This is what we need for AI development as well: A framework or model that by default returns into a safe state and does not escalate harmfulness. How that might look like I don’t know, maybe Emotional Models that intervene with the pure logic based algorithms are a solution.

But as long as it is actually easier to write harmful AIs, it is only a matter of time until something goes wrong.

[Via Accelerating Future Blog]

Stumble it!

Popularity: 21%