Intelligence, Consciousness, and Motivation
When we think about computers becoming more intelligent, it is natural to begin to ascribe to them human characteristics. Homo sapiens is the most intelligent machine we have experience of, and we are used to thinking of other machines as becoming more human as they become more intelligent.
Such thinking, however, has lead us to conflate a number of different characteristics that ought to be regarded as separate. When we imagine a machine that meets or surpasses human intelligence we often also imagine it being conscious and having desires.
This is a problem, because these two additional characteristics are not necessitated by intelligence alone. At least on the face of it, it seems possible to have a highly intelligent machine that is not conscious and does not have desires. Our natural tendency to assume human characteristics for intelligent machines is obfuscating our ability to think clearly about the future of civilisation.
Much discussion in this area has focused on the question of what super intelligences will ‘want’. A lot of popular sci-fi has asked this question, and has frequently answered with rather far-fetched apocalyptic scenarios. Of course, we should not take such entertainment too seriously, but we do not seem to be doing all that much better elsewhere. Much intelligent discussion within the transhumanist and singularitarian community has fallen into the trap of assuming human characteristics for AGIs. A recent post over at the Suzanne Gildert’s fantastic Physics and Cake asked ‘What do superintelligences really want?’ Gildert made some excellent points, but claimed
In fact, if our superintelligent program has no hard-coded survival mechanism, it is more likely to switch itself off than to destroy the human race willfully.
I believe this is based on flawed thinking. None of today’s computer programs have ‘hard-coded survival mechanisms’, but they do not shut themselves off. If all we know about a program is that it is super-intelligent, what guesses can we make about how it will act when first turned on? My guess is that it will do absolutely nothing. Why should it do anything else? We have no reason to think such a machine would be conscious or driven to do anything. Until we give it motivation to act, it will sit idle. Even switching itself off is an act that requires explanation and motivation. What would drive the machine to behave in such a way?
The article then considers how we might build motivation into our computer intelligences. This is an excellent question, but Gildert worries unnecessarily that any such scheme to control the motivation of an AGI will be undermined by its ability to control its own code. Won’t such a being simply alter its reward mechanisms to suit its own ends?
Again, this is based on a flawed comparison with humans. Humans are complex systems, whose reward structures have evolved in many stages over millions of years. The ancient ‘lizard brain’ has one set of instincts that were designed to promote survival in a brutal world. But the more recent prefrontal cortex permits far more complex modelling of the environment. While the old animal instincts were great survival rules of thumb—giving the best rule to follow for most circumstances—the new models allow a much richer and more intelligent range of responses. But the old instincts are still there, and frequently conflict with ancient instinctive behaviour.
Humans think that super-intelligent computers might want to override their reward mechanisms because that is what humans often desire. Who would not want to be able to artificially control his desire for food, or his ability to concentrate for extended periods? But these are just examples of two parts of the brain conflicting. The old lizard brain against the new higher reasoning centres.
No such conflict exists for computers. To think that an AGI might override its motivational code in order to suit its own desires assumes that it has other desires. This is nonsense. Where do these desires come from? The only motivations a super-intellignce will have are those we give it.
