About 9 years ago, on June 10th 1999, the first humans fell victim to a computer control system failure. Due to a chain of unfortunate circumstances, a the pressure on a 16-inch steel pipe line increased uncontrollably and caused it to rupture, flooding two small creeks near Bellingham in the process. The petrol ignited and killed two 10-year-old boys and an 18-year-old adolescent. Although it was later revealed that the accident was in part caused by human failure, it is still considered as the first cyber-event that killed humans.

Now, you might think that things have changed since then, but you’d be wrong. Joe Weiss from Applied Control Solutions says:

Until eight years ago, my whole life was making control systems usable and efficient, and, by the way, very vulnerable. It is exactly what you will find today in many, many industrial applications. This isn’t just 1999. No, this is June 2008.

And it is true. The computer systems that control complex technical facilities, like the power grid, traffic control infrastructure, or pipeline systems have historically only been optimized with respect to usability, efficiency and effectivity. Security has almost never been a concern in the design of these systems. Many of them have been developed before the rise of the internet, so that attacks had to be carried out physically. But eventually some systems responsible for the control of important infrastructure have been connected to a network, often in an effort to improve control or just because of company policies, without any evaluation of possible risks.

There are more examples. The 787 Dreamliner from Boeing, which is scheduled to come out later this year, may be vulnerable to hacker attacks. This is because the flight control network is separated from the passenger network only by software firewalls, and thus might be attacked from inside the plane (or even from the outside, with the help of a trojan horse maybe).

When military control systems fail, it gets nasty quite quickly. Last year, a robot anti-aircraft cannon killed 9 and wounded 14 soldiers when it went berserk during a training of the South African National Defence Force. However, it is not yet clear if the accident was caused by a mechanical malfunction or software failure.

And it is not only the security of control systems that poses a risk, the control systems themselves are feared by researchers to remove human control from key decision-making processes. Tom Rodden, Professor of Interactive Systems at the University of Nottingham concluded at the Human-Computer Interaction conference that this loss of control might not directly be a risk for our lives, but surrenders “basic human values and concepts such as personal space, society, identity, independence, perception, intelligence and privacy.”

This lack of security must become a higher priority to control system designers, especially as there is an unavoidable trend towards more autonomous, intelligent systems that can cope with the size and complexity of modern infrastructure. We have to take care that we do not hand over all control of critical systems and uncritical but socially important concepts to machines. Even if a worst case scenario - such as malicious AI breaking loose - does not happen, programs are likely to be susceptible to bugs and misinterpreted signals, and therefore have to be very carefully designed. As the examples in the past have shown, these issues are a real threat to human beings and the lack of security against tampering from the outside increases the AI Panic level by +1%.

Stumble it!

Popularity: 29%