AI Panic!

I’ve written about AI in video games and their real-life impact a while ago, with the conclusion that it is quite basic and does not really contain “intelligence”. Now with all the hype around the newest version of the Grand Theft Auto series, GTA IV, and a development budget of about $100 million, lets take a closer look if anything has changed.

GTA is basically a sandbox game with a good story tucked on. As a gangster in a virtual city, this iteration it is Liberty City, modelled after New York, you are free to roam around in all kinds of vehicles, from bikes, lots of cars up to military helicopters and do all kinds of things - most of them illegal, of course. Everything that is forbidden in the real world becomes not only possible, but a goal in GTA, from assassination tasks, drug trafficking, gang wars and - of course - car theft there’s everything there what the villain’s heart desires (I wont go into the psychological aspects here, but it is a real concern for parents and politics, especially the German government is pressing hard to ban Killerspiele).

There is a wide variety of actions in Liberty City, especially when it comes to driving and destroying things. Also, such a city simulation requires believable and complicated (artificial) agent behaviour, to allow the player to suspend his disbelief. And in fact, it is an interesting and often funny pastime to just watch non-player character (NPC) interactions, their discussions and police activity.

The general pedestrian behaviour and driving AI has improved considerably over GTA: San Andreas, but there are still the odd situations where agents don’t react properly. Some mention that the cover seeking behaviour of the police has some glitches, so that policemen in search for a new cover run around the side of the police car facing you instead of the back, or that they fall off buildings when you hide just behind the edge. However, these situations are rare and are often more funny than annoying. The AI, while not perfect, does it’s job alright. All the situations where you say “Wow, I didn’t think it would be that smart” are most likely scripted though, meaning that they have been predefined by the game designers and are not (fully) dynamically generated. Another indicator for the limitations of the AI system is the density of pedestrians, or rather, the lack of it. Similar to the previous incarnations, there are not nearly enough people or cars in the streets. Admittedly, this increase in crowd density is hard to implement correctly and is probably an effect of limited CPU power.

GTA4 is a good example of fake AI, artificial intelligence that only pretends to reason. This is very common in video games, as their primary goal is not to plan realistically and beat the player, but to provide enjoyment and make (smart) mistakes. This becomes apparent in how the game handles the appearance of police units. When your wanted-level increases, there are suddenly more police cars and cops around (especially for the higher levels), that come pouring out from behind street corners or are just there when you turn around. This behaviour, while perfectly legit for a game, would be useless in real life. All the steps which are most important for navigation and reasoning in robots and “proper” AI are cropped away in video games. Therefore there is -still- no possibility that non-player villain behaviour (would that be the police then, in GTA IV?) accidentally or even on purpose crosses over into the real world from a game AI.

Side note: As you may have noticed, I didn’t have a lot time to update AI Panic very often over the last weeks. At the moment I’m quite busy at the university, I have to hand in a kind-of-important internal report in about a month of what I’m planning to do in the next 2-3 years. Once this is out of the way, I’ll have more time for the blog again. Sorry about that!

About 9 years ago, on June 10th 1999, the first humans fell victim to a computer control system failure. Due to a chain of unfortunate circumstances, a the pressure on a 16-inch steel pipe line increased uncontrollably and caused it to rupture, flooding two small creeks near Bellingham in the process. The petrol ignited and killed two 10-year-old boys and an 18-year-old adolescent. Although it was later revealed that the accident was in part caused by human failure, it is still considered as the first cyber-event that killed humans.

Now, you might think that things have changed since then, but you’d be wrong. Joe Weiss from Applied Control Solutions says:

Until eight years ago, my whole life was making control systems usable and efficient, and, by the way, very vulnerable. It is exactly what you will find today in many, many industrial applications. This isn’t just 1999. No, this is June 2008.

And it is true. The computer systems that control complex technical facilities, like the power grid, traffic control infrastructure, or pipeline systems have historically only been optimized with respect to usability, efficiency and effectivity. Security has almost never been a concern in the design of these systems. Many of them have been developed before the rise of the internet, so that attacks had to be carried out physically. But eventually some systems responsible for the control of important infrastructure have been connected to a network, often in an effort to improve control or just because of company policies, without any evaluation of possible risks.

There are more examples. The 787 Dreamliner from Boeing, which is scheduled to come out later this year, may be vulnerable to hacker attacks. This is because the flight control network is separated from the passenger network only by software firewalls, and thus might be attacked from inside the plane (or even from the outside, with the help of a trojan horse maybe).

When military control systems fail, it gets nasty quite quickly. Last year, a robot anti-aircraft cannon killed 9 and wounded 14 soldiers when it went berserk during a training of the South African National Defence Force. However, it is not yet clear if the accident was caused by a mechanical malfunction or software failure.

And it is not only the security of control systems that poses a risk, the control systems themselves are feared by researchers to remove human control from key decision-making processes. Tom Rodden, Professor of Interactive Systems at the University of Nottingham concluded at the Human-Computer Interaction conference that this loss of control might not directly be a risk for our lives, but surrenders “basic human values and concepts such as personal space, society, identity, independence, perception, intelligence and privacy.”

This lack of security must become a higher priority to control system designers, especially as there is an unavoidable trend towards more autonomous, intelligent systems that can cope with the size and complexity of modern infrastructure. We have to take care that we do not hand over all control of critical systems and uncritical but socially important concepts to machines. Even if a worst case scenario - such as malicious AI breaking loose - does not happen, programs are likely to be susceptible to bugs and misinterpreted signals, and therefore have to be very carefully designed. As the examples in the past have shown, these issues are a real threat to human beings and the lack of security against tampering from the outside increases the AI Panic level by +1%.

Imagine, someday, we’ll lose the control over our computers to some sentient being, unable to stop it or to unplug the millions of home pcs, office workstations or lab mainframes affected.

Well, there is no need to imagine, this very thing has been happening for a long time now, in the form of Zombie Computers. These machines have been compromised by hackers, viruses or trojans, and are remote controllable. Often they are grouped in a network of zombie computers, called a botnet.

The RSA 2008 conference in San Francisco earlier this month had a panel discussion on this topic, and the tone was “a mix of resignation, indignation and post-9/11 rhetoric”, according to Threat Level. According to the panelists, botnets are the biggest threat in the internet today and a danger to national security.

However, little is done to combat the use of zombie computers. Although some botnet operators are caught once in a while, the spam and attacks they create does not seem to decrease. I agree with Ira Winkler, a security consultant who said at the conference:

“The problem is no one is doing anything. Guess what? If your system has a bot on it, you don’t get on the internet. [...] We need to hold people responsible when they present an imminent threat to other people.”

However, ISPs are reluctant to cut off offenders, as they often have no clue what’s going on and respond with confusion and angry phone calls, which drive up the customer support costs.

The inability to quickly and efficiently stop these bot nets could prove very helpful for a malevolent AI. Once released on the internet (or on its successor, maybe The Grid), there is virtually no way to stop it again. The infrastructure to spread uncontrollably is already given by the botnets. Unless there is some fundamental change on the network level and software level (hello Microsoft!), the only way to stop such an outbreak would be the complete shut-down of the net. Of course, this would cause quite a disturbance to the world economy, as most financial transactions and international trading are entirely done online. Most nations will be hesitant to risk that, just because some strange program has been set free from a university lab somewhere.

Therefore I think the current infrastructure of the web eases the spread of an AI (Panic Level: +3%), once it discovers the weaknesses of connected hosts and “understands” how botnets work. But that’s not yet an immediate threat; after all we have to develop the artificial general intelligence first, which, mind you, is a little tricky.

If you think that artificial intelligences that actively attack and try to kill its human counterparts are just a hypothetical idea that might never come to reality, you might be mistaken.

The gaming industry has been training gruesome robots, mighty automated armies and deadly opponents for years now. So one might wonder, how dangerous are these algorithms really?

I was inspired to write a bit about this topic by a comment from Horus Aha on last weeks article about Ray Kurzweil’s talk on the Future in Gaming at the Games Developer Conference:

Lots of investment in making game AI’s that are basically sociopathic killers explicitly designed to effectively compete against humans. This applies in 1st person shooters, but also to other genres. The games are networked and it is possible that the AI’s in these games may learn how to game-the-game by finding ways to compete with players outside of game environments. Why not? I don’t think anybody would have taken warnings about “gold-farming” seriously until it was already a reality.

Now, is this fear justified? Are we really endangered by computer game AI?

No.

AI in the entertainment industry is incredibly dumb. The are a couple of reasons for that. Probably the most important is that the game developers do not want an intelligence that can out-reason humans and win all the time: The goal is making an AI that plays to lose, not to win. It is a well-known psychological fact that an opponent that almost beats but eventually loses against the player is most enjoyable for him (There was a good presentation about that from Soren Johnson at this year’s GDC called Playing to Lose: AI and “CIVILIZATION”).

But don’t we need a proper AI to be able to figure out how to achieve this? When you look at the concepts of AI in games and talk to game developers it quickly becomes clear that they don’t even try to make a learning or guessing AI: They cheat! Faking intelligence is a lot easier than actually achieving proper reasoning. In almost all games with incomplete information, i.e., a fog of war or hidden objects, the AI is given full information and only pretends to not know where you are and what you do. And if you see a really smart manoeuvre from the enemy troops (or your virtual team-mates) it is hard-coded or pre-scripted very often. I attended the AI roundtable talks at the GDC in San Francisco two weeks ago and almost none of the attending developers (who worked on Bioshock, Halo3, The Sims, Command & Conquer etc.) has used any of the “proper” academic AI algorithms like neural networks or genetic algorithms.

There are some exceptions, for example Black & White uses a simple reinforcement learning algorithm to make the creature learn from your actions. F.E.A.R. is an often cited example of real planning in a game, however, the used algorithm is really simple and some argue that the resulting behaviour could have been achieved with pre-written scripts just as well.

So there’s no real, potentially dangerous, intelligence in games at all. This is why the AI Panic Level drops by -5%, the “sociopathic killers” will not be able to escape the virtual world of computer games for a long time. If at all, they will become smart only by developments outside the games industry, for example academic research.

The Gulf Cooperation Council might have to rely on artificial intelligence technologies to fight unemployment, GulfNews.com reports. Joachim Diedrich, a professor of artificial intelligence at the Uni of Queensland, Australia, embraces this dependency:

“For decades now, there has been a misconception that artificial intelligence will replace human resource. Rather, it is a job creator. [...] I see this as a positive opportunity. We should be optimistic about the availability of new jobs through the development of artificial intelligence.”

Probably we should indeed be optimistic, as there might not be a way around a tighter integration of AI into our daily life and products anyway. This integration has long started, it is being used in predicting weather and the stock market, so there is no way (and use) of denying its importance to the job market.

Although Diedrich tries to allay security doubts by saying that he does not think that “the industry will come to depend on artificial intelligence,” he lowers the credibility of his own statement with his next: “The use of artificial intelligence is not public knowledge, we don’t know how or where it is being used in the region.”

This far-reaching integration of black-box AI into important economical factors like the stock exchange and a huge number of jobs creates a dangerous dependency with regards to the worst case scenario of a hostile AI (it increases the AI Panic Level by +2%).

While current AI methods, used for example to predict the weather or stock values, are quite simple and pose no threat whatsoever, it will get more and more difficult to say that of future techniques. The accelerating technological advancement will enable very complex systems that incorporate extensive background knowledge from news articles, common sense databases and historical data to optimise the accuracy of predictions. This accurate world model might enable AIs to reason about everyday life and is a requirement for general artificial intelligence.

Thus a high job dependency on AI in this area will feed into the advancement and research of this area and brings us closer to a possibly dangerous strong AI.

At the “Up Experience” conference in Houston, Steve Wozniak had a very pessimistic view on how smart robots are and how long it will take them to become intelligent. In an interview with the Houston Chronicle, he said:

“I don’t think we’re close. I don’t think we’ve gone one step. [...] I think we understand a bare amount of what the mind does.”

A robot that could simulate human life would have to be around for eight or so years before it could have a child’s understanding, he noted.

Steve Wozniak sees the day robots become intelligent (which is, in his eyes, be able to get him a cup of coffee) still far away in the future. Although this opinion could be classified into Kurzweils “linear thinking” category, Wozniak is undeniably an expert in the technology sector. Therefore the AI Panic Level will be reduced by -1%.

A new study by ABI Research shows that by 2015 the market for personal robotics components might reach $12 billion. They expect task robotics (that ease routine chores like cleaning, driving, etc) and entertainment robotics to have roughly equal parts in this revenues.

The computerworld article goes on to speculate that a part of the entertainment sector will be made up by human-like robots that can act as companions (see also my article on seducing robots for more on this).

By 2015, ABI Research expects more than 21 million robots shipped yearly, and predicts very high demand in sensors, actuators, servos and manipulators.

This study indicates that elaborate robots will penetrate the market very soon, which is only the next step after “intelligent” microcontrollers, that can nowadays be found everywhere, in cars, toys and even rice cookers. This increased dependency on machines with a high level of computerisation and connectivity together with a lacking sense for the possible threats of advanced AI is dangerous, the AI Panic Level increases by +3%. In the worst case scenario (which we have to take care of) robots with a network connection could be remotely controlled or reprogrammed (for example through a tampered firmware update) to follow the commands of an AI that went berserk.

Sure, today’s rice cookers are really a helpful invention, especially to cultures with a high rice dependency factor. So it is only logical to apply the newest AI technology to make cooking rice even more exciting and simple. According to The Edge Daily that’s exactly what Philips did, and their new flagship rice cooker “features automatic artificial intelligence controls that enables it to cook a variety of rice recipes including claypot rice, cakes, congee and soups at the touch of a button.”

Philips did not mention a behavioral model that activates when the owner fails to clean her rice cooker or neglect its needs otherwise, the cooker subsequently adds an undisclosed substance to the rice, to keep women who are “housewives or aspire to be great housewives” in line.

This thoughtless integration of AI into commonplace appliances might be another step to AI world domination. As we’ve shown recently, the male population will be kept in check by love-robots. Now Philips delivers the solution for the remaining half of humankind, enslaved by evil rice cooker overlords women will be forced to work in the rice cultivation industry.

No, really, that’s how it’s going to happen!

The US company DIAGNOS has made a deal with US Gold to use its technologies in the search for natural gold and silver reserves in Mexico. US Gold is paying $230,000 CDN for this service and will get some juicy bonuses if some precious metals are actually found. The “AI” behind the technology DIAGNOS applies, however, is quite a straight forward approach, in which maps (satellite images, topography, geology, etc.) are weighted and combined into a “prospectivity map”.

This technology might be used to dig holes, but is not ground breaking on the AI side. Statistical learning has been around in for quite a while now and there is lots of research on it. However, it is nice to see some of that research actually applied for useful activities, like exploiting natural resources. The slow speed in which actual AI research results are adopted by commercial companies (possibly with the exception of computer vision, i.e., face recognition) can be linked to the often difficult to understand algorithms and almost always impossible to predict outcomes of them. This inherent complexity of AI algorithms (especially of those that aspire world domination!) prevents fast adaptation by the industry and thus limits research to academia somewhat.The algorithms used by DIAGNOS are no exception of that. There is no danger of them (or any other similar algorithm for that matter) becoming conscious, thus no need to panic: As an example of state of the art applied AI, it shows how far we still are from doomsday events, therefore the AI Panic Level changes by -0.1%.