Author Topic: Conscious Machines  (Read 1513 times)


  • Administrator
  • Sr. Member

  • Offline
  • *****
  • 299
    • View Profile
    • Higher Intellect
Conscious Machines
« on: February 11, 2017, 04:21:06 pm »
Conscious Machines

Marvin Minsky

Published in "Machinery of Consciousness", Proceedings, National Research
Council of Canada, 75th Anniversary Symposium on Science in Society, June
1991. I don't have the final publication date.

Many people today insist that no machine could really think. "Yes," they
say, "machines can do many clever things. But all of that is based on
tricks, just programs written by people to make those machines obey
preconceived rules. The results are useful enough -- but nowhere in those
cold machines is there any feeling, meaning, or consciousness . Those
computers simply have no sense that anything is happening."

They used to say the same about automata vis-a-vis animals. "Yes, those
robots are ingenious, but they lack the essential spark of life." Biology
then, and psychology now: each was seen to need some essence not mechanical.

The world of science still is filled with mysteries. We're still not sure of
how the Sun produces all its heat. We do not know precisely where our early
ancestors evolved. We can't yet say to what extent observing violence leads
to crime. But questions like those do not evoke assertions of futility. We
can try harder to detect more neutrinos, find more fossils, or perform
throrough surveys. However, in certain areas of thought, more people take a
different stance about the nature of our ignorance. They proceed to work
hard, but not toward finding answers, but toward trying to show that there
are none. Thus Roger Penrose's book [1] tries to show, in chapter after
chapter, that human thought cannot be based on any known scientific

I have already written a book [2] that discusses various attempts to show
that men are not machines, but mainly works to demonstrate how the contrary
might well be so. You might object that no one has time to read all such
books, so why can't I just summarize? And that's what this essay is all
about: that certain things are too complex to summarize! This includes the
mechanisms of highly evolved organisms and, especially, the workings of
their nervous systems. It also includes the highly evolved systems that we
call cultures and societies. And especially, it includes what we call
consciousness. In particular, consider the problem of describing the brain
in detail -- in view of the fact that it is the product of tens of thousands
of different genes. We can certainly see the attractiveness of proposing to
get around all that stuff, simply by postulating some novel "basic"
principle by which our minds are animated by some vital force or essence we
call Mind, or Consciousness, or Soul.

That tendency is not confined to religion and philosophy. The same approach
pervades our everyday psychology. We speak of making decisions by exercising
'freedom of will'; or by finding what something 'means', or of discovering
truths by means of 'intuition'. But none of those terms explains very much;
each only serves to name another set of mysteries.

The situation is different in Physics. Consider the whirlpools that form
when water flows down drains. When a scientist says that this can be
explained by the Conservation of Momentum, that's very different from
attributing it to some convenient Whirlpool God -- because precisely the
very same mathematical rule can be used to explain and predict a vast range
of other phenomena, with a precision and lack of exception found in no other
realm of ideas. That principle apparently applies to everything in our
universe and, because of its singularly good performance, we regard this
sort of "fundamental" or "unified" principle as an ideal prototype of how to
account for mysterious phenomena. But one can carry that quest too far by
only seeking new basic principles instead of attacking the real detail. This
is what I see in Penrose's quest for a new basic principle of physics that
will account for consciousness.

The trouble is that this approach does not work well for systems whose
behavior has evolved through the accretion of many different mechanisms,
over the course of countless years. For example, in physiology, the
excretion of excess potassium in the urine occurs because our ancestors
evolved elaborates system of receptors and transport mechanisms, along with
intricate machinery for controlling them. This is understood so well today,
that no one feels that there's any need to postulate a separate, special
principle for the Conservation of Potassium. Progress in this area is no
longer news for biology because we have seen two hundred years of great
success accrued from working out details. Since Harvey, Darwin and Pasteur,
the idea of a Vital Force has nearly vanished from biology. Why is it still
so much a part of present-day psychology?

I'll argue that vitalism still persists because we're only starting to find
a way to understand the brain. (I see this as the irony of Penrose's book,
because the path toward understanding lies in that flood of new ideas that
began to grow around the time, half a century ago, along with the emergence
of computers in the 1950s -- include the work of Turing in 1936, McCulloch
and Pitts in 1943, and the hundreds of thinkers who joined them afterward.
Yet Penrose takes the other side, and argues that the abilities of human
mathematicians to discover new mathematical truths cannot be explained on
the basis of anything a machine could do. He argues in [1], p110, that this
kind of thinking must be based on "insights that cannot be systematized --
and, indeed, must lie outside any algorithmic action!" He bases this on the
assumption that any thinking machine we build for attempting to discovering
knowledge about mathematics must itself be based on some absolutely
consistent logical foundation -- that is, one that cannot possibly produce
any type of logical contradiction or inconsistency. This is the same
assumption used in Godel's celebrated 'incompleteness theorem'. Penrose's
application of this idea to psychology is due, as Penrose notes, to
J.R.Lucas, in Philosophy, 36, pp120-4, 1961.


It seems to me that all of this stands upon a single and simple mistake. It
overlooks the possibility, as my colleague Drew McDermott once remarked, of
including systems "that are mistaken about mathematics to some degree, or
systems that can change their minds." By inadvertently ruling such machines
out, you've simply begged the question whether human mathematicians can be
kinds of machines -- because people do indeed change their minds, and can
indeed be mistaken about some parts of mathematics. An entire generation of
logical philosophers has thus wrongly tried to force their theories of mind
to fit the rigid frames of formal logic. In doing that, they cut themselves
off from the powerful new discoveries of computer science. Yes, it is true
that we can describe the operation of a computer's hardware in terms of
simple logical expressions. But no, we cannot use the same expressions to
describe the meanings of that computer's output -- because that would
require us to formalize those descriptions inside the same logical system.
And this, I claim, is something we cannot do without violating that
assumption of consistency.

If you are not a logician, then you might wonder what's all the fuss about.
"What could possibly be wrong with logical consistency. Who wants those
contradictions, anyway?" The trouble with this is that the problem is worse
than it looks: paradoxes start to turn up as soon as you permit your machine
to use ordinary common-sense reasoning. For example, troubles appear as soon
as you try to speak about your own sentences, as in "this sentence is false"
or "this statement has no proof" or in "this barber shaves all persons who
don't shave themselves." The trouble is that when you permit "self
reference" you can quickly produce absurdities. Now you might say, "Well
then, why don't we redesign the system so that it cannot refer to itself?"
The answer is that the logicians have never found a way to do this without
either getting into worse problems, or else producing a system too
constrained to be useful.

Then what do ordinary people do? So far as we know they scarcely use any
logic at all. The studies made by the great child psychologist Jean Piaget
suggest that the abilities required for to manipulating formal expressions
are not reliably available to children until their second decade, if ever.
And even as a mathematician, I cannot recognize the psychology Penrose
describes. When doing mathematics, my mind is filled with many things
non-logical. I imagine examples based on gears and levers, I imagine
conversations that might reveal to me what Andrew Gleason or Dana Scott
might do in the same situation, or I imagine explaining my solution to a
student and discovering something wrong with it There's little sign of
consistency in any of that experience. Nor is that famous 'intuition' really
a privileged route to the truth, because although the answer seems to come
with a feeling of certainly, later it's likely to turn out to be wrong.

Perhaps the most important aspect of how humans work are the ways in which
we ask ourselves (not necessarily by using words) what problems have we seen
before that most closely resemble the present case, and how did we manage to
deal with them. For those were where we made our mistakes and then sometimes
managed to learn from them. And notice that in doing so, we somehow must
employ some capabilities for retrieving and then manipulating some
descriptions of some of our earlier mental activities. Now, notice how
self-referent this is. Often when you work on a problem you consider doing
some certain thing -- but before you actually carry that out, you often
inquire about yourself, about whether you actually be able to carry it
through. Solving problems isn't merely applying rules of inference to
axioms. It involves making heuristic assessments about which aspects of the
problem are essential, and which of one's own abilities might be adequate to
dealing with them. Then, whatever happens next arouses various feelings and
memories about of situations that seem similar and methods that might be
appropriate. Is this done by some kind of non-physical magic or it is
accomplished, as I maintain, by the huge and complex collection of
knowledge-base representations and pattern-matching processes that we all
regard as 'common sense'?

Now it happens that when we do such things, we often find that we talk to
ourselves about what we're doing. And when we thus "refer to ourselves" we
sometimes speak of being conscious or aware. I think it no coincidence that
Penrose feels that this, too is something present-day science cannot
explain. Indeed, he Could this result from just that fear of inconsistency
and self-reference? Indeed, Penrose sometimes speaks of a "reflection
[principle" with something resembling awe: "The type of 'seeing' that is
involved in a reflection principle requires a mathematical insight that is
not the result of the purely algorithmic operations that could be coded into
some formal mathematical system (p110)." In my opinion this is just a
mistake! He appears to assume that when this is applied to humans, the word
"consistent" can be freely inserted between 'some' and 'mathematical' -- as
though people possess some marvelous gift whereby they can tell which
assertions are true. But in view of the many mistakes we all make, I see no
compelling evidence that anyone has any direct such access to truth. All we
can depend upon (including the power of formal proof) is based on our
experience. I think. And in any case there really is no problem at all in
programming a computer to perform that sort of reflective operation. Indeed
John McCarthy has pointed out that forming a Godel sentence from a proof
predicate expression (which is the basis of the Lucas-Penrose argument)
requires no more than a one line LISP program. So in my view Penrose and
many other philosophers have put the problem upside down: the difficulty is
not with making algorithms that can do reflection -- which is easy for
machines, but with consistency -- which is hard for people. In summary,
there is no basis for assuming that humans are consistent -- not is there
any basic obstacle to making machines use inconsistent forms of reasoning.


Even the most technically, sophisticated people maintain that whatever
consciousness might be, it has a quality that categorically places it
outside the realm of science, namely, a subjective character that is makes
it utterly private and unobservable. Why do so many people feel that
consciousness cannot be explained in terms of anything science can presently

Instead of arguing about that issue, let's try to understand the source of
that skeptical attitude. I have found that many people maintain that even if
a machine were programmed to behave in a manner indistinguishable from a
person, it still could not have any subjective experience. Now isn't that a
strange belief -- considering that unless you were a machine yourself, how
could you possibly know such a thing? As for 'subjectivity,' consider that
talking about consciousness is a common, objective form of behavior.
Therefore, any machine that suitably simulated a human brain would have to
produce that behavior. Then, wouldn't it be curious for our artificial
entity to falsely claim to have consciousness? For if it had no such
experience, then how could it possibly know what to say? Of course a classic
question in philosophy is asking for proof that our friends have minds;
perhaps they are merely unfeeling machines. But then one must ask how they'd
know how to lie.

In any case, we have much the same problem with ourselves; try asking a
friend to describe what having consciousness is like. Good luck! Most likely
you hear only the usual patter about knowing oneself and being aware, of
sensing one's place in the universe, and so on. Why is explaining
consciousness so dreadfully hard? I'll argue that this is something of an
illusion, because consciousness is actually easier to describe than most
other aspects of mind; indeed, our problem is a far more general one,
because our culture has not developed suitable tools for discussing and
describing thinking in general. This leads to what I see as a kind of irony;
it is widely agreed that there are "deep philosophical questions" about
subjectivity, consciousness, meaning, etc. But people have even less to say
about questions they'd consider more simple:

        How do you know how to move your arm?
        How do you choose which words to say?
        How do you recognize what you see?
        How do you locate your memories?
        Why does Seeing feel different from Hearing?
        Why does Red look so different from Green?
        Why are emotions so hard to describe?
        What does "meaning" mean?
        How does reasoning work?
        How do we make generalizations?
        How do we get (make) new ideas?
        How does Commonsense reasoning work?
        Why do we like pleasure more than pain?
        What are pain and pleasure, anyway?

We never discuss these in everyday life, or bring them up in our children's
schools. An alien observer might even conclude that those Earth-people seem
to have a strong taboo against thinking about thinking. It seems to me that
this is because our traditional views of psychology were so mechanistically
primitive that we simply had no useful ways to even begin to discuss such
things. This is why I find such irony in the arguments of those who reject
the new mechanistic concepts of psychology -- the new ideas about
computational processes that promise at last to supply us with adequate
descriptions of these complex processes.

The science of Psychology, as we know it today, is scarcely one hundred
years old. Why did humanity wait so long before the emergence of thinkers
like Freud, Piaget, and Tinbergen? I think the answer lies in the fact that
the brain is not merely a kind of machine, but one that is far more complex
than anything ever imagined before. The pivotal notion provided by those
three pioneers was that the mind has many parts. A person doesn't simply See
by "looking out" through the eyes. Instead, vision involves many different
processes, cooperating, competing, being promoted and inhibited by other
processes, being managed and regulated by yet others. You can not simply
'recognize' a telephone, because that is scarcely at all a matter of vision;
instead, you have to "re-cognize" it -- that is, the input has to somehow
activate some memory representations of a device with a certain kind of
structure (handset and dial, say) coupled with a certain functional
disposition (to hold to the mouth and ear for communication purposes). This
is nothing like the sorts of unitary concepts found both in commonsense and
philosophy, e.g., of a platonic ideal of a telephone, or some sort of model
inside the head. In recent years we've learned much more about the
complexity of the brain. It now appears that perhaps fully half of our
entire genetic endowment is involved in constructing our nervous systems.
This would suggest that the brain is nothing like a single large-scale
neural net; instead, it would have even more parts than the skeletomuscular
system -- which can be seen to have hundreds of functional parts. If you
examine the index of a book on neuroanatomy, you will find the names of
several hundred different organs of the brain. A good fraction of those are
already known to have psychologically distinct functions. To pursue the
analogy a little further, note that the skeletal anatomies of animals have
been known for millennia, but only in rather recent years have scientists
understand the mechanics of locomotion and its various gaits; that had to
wait until scientists learned more about the mechanics of forces and
materials. Similarly, mechanistic theories of psychology may have to wait
even longer for adequate conceptual tools because the 'mechanics' of
heuristic computation could turn out to be more complex than those of
physics. Before these new ideas emerged, with the era of complex
information-processing computer models, such models were not considered
convincing -- perhaps because there were no feasible experiments. I don't
mean to say that there was no progress at all before computers, only that
there was precious little. Freud himself was one of the first to conceive of
"neural-net-like" machines -- only no one would listen to him except Fliess.
Later came the astounding insights of Post, Godel, and Turing, followed by
those of Rashevky's group, McCulloch and Pitts, and Grey Walter's simple yet
somewhat life-like mini-robots. But significant progress began only in the
1950s when more serious models could be conceived, tested, and discarded in
days or weeks instead of years. Soon the researchers in Artificial
Intelligence discovered a wide variety of ways to make machine do pattern
recognition, learning, problem solving, theorem proving, game-playing,
induction and generalization, and language manipulation, to mention only a
few. To be sure, no one of those programs seemed much like a mind, because
each one was so specialized. But now we're beginning to understand that
there may be no need to seek either any single magical "unified theory" or
and single and hitherto unknown "fundamental principle"-- because thinking
may instead be the product of many different mechanisms, competing as much
as cooperating, and generally unperceived and unsuspected in the ordinary
course of our everyday thought.

What has all this to do with consciousness? Well, consider what happened in
biology. Before the 19th century there seemed to be no alternative to
concept of "vitality" -- that is, the existence of some sort of life-force.
There simply seemed no other way to explain all the things that animals do.
But then, as scientists did their work, they gradually came to see no need
for a "unified theory" of life. Each living thing performed many functions,
but is slowly became clear that each of them had a reasonably separate
explanation! For the most part each separate function was served by a
different and specialized organ! Thus the lungs oxygenate the blood, while
the heart pumps it to other organs. The nucleus reproduces all the organs'
structural information, while the ribosomes translates those codes into
proteins which then self-configure themselves. For some time that subsequent
appeared to entail a mystery. It seemed natural to assume that those
configurations were based on a uniform energy-minimizing mechanism -- but
simulations did not bear this out. This appears to not be so; instead, each
protein has had to evolve this property on its own. (A random string of
peptides cannot usually manage it.) Conclusion: There is no central
principle, no basic secret of life. Instead, what we have are huge
organizations, painfully evolved, that manage to do what must be done by
hook or crook, by whatever has been found to work.

Why not assume the same for the mind? (I could have said the brain, instead
-- but in my view minds are simply what brains do.) Why else would our
brains contain so many hundreds of organs? Of course there are many old
arguments against localization of brain-functions because it seemed that
often a mind still works when some of its brain has been lost. One answer to
that is to argue that many functions are accomplished in multiple ways, not
only to provide resistance to some injuries, but perhaps more important,
because no particular way is likely to be always reliable. To be sure, there
still seem to be some mental phenomena that have not yet been shown to
"organ-ized". So there is still some room for theories about mechanisms that
are not so localized. But now, I maintain, it is time for "insulationism" to
take its place along with, and in complementary opposition to,

Then what might be the functions and the organs of what we call
consciousness? To discuss this, we'll have to agree on what we're talking
about -- so I'll use the word consciousness to mean the organization of
different ways we have for knowing what is happening inside your mind, your
body, and in the world outside. Here is my thesis; some people may find it
too radical:

We humans do not possess much consciousness. That is, we have very little
natural ability to sense what happens within and outside ourselves.

In short, much of what is commonly attributed to consciousness is mythical
-- and this may in part be what has led people to think that the problem of
consciousness is so very hard My view is quite the opposite: that some
machines are already potentially more conscious than are people, and that
further enhancements would be relatively easy to make. However, this does
not imply that those machines would thereby, automatically, become much more
intelligent. This is because it is one thing to have access to data, but
another thing to know how to make good use of it. Knowing how your pancreas
works does not make you better at digesting your food. So consider now, to
what extents are you aware? How much do you know about how you walk? It is
interesting to tell someone about the basic form of biped locomotion: you
move in such a way as to start falling, and then you extend your leg to stop
that fall: most people are surprised at this, and seem to have which muscles
are involved; indeed, but few people even know which muscles they possess.
In short, we are not much aware of what our bodies do. We're even less aware
of what goes on inside our brains.

Similarly we can ask the extents to which we're aware of the words we speak.
At first one thinks, "yes, I certainly can remember that I just pronounced
"the words we speak." But to what extent are we aware of the process that
produced those particular words? Why, barely at all! We have to employ
linguists for lifetimes of research even to discover the simplest aspects of
the language production process.

Finally, I can ask you questions like, "Can you tell me what you are
thinking about." The answers to such questions are hard to interpret. The
listener might list the names of some subjects or concerns that were
recently in mind, and sometimes can describe a bit of the trains of thought
that led to them. These kinds of answers clearly feed upon memories of
recent brain-activities. But every such answer seems incomplete, as though
the act of probing into any one of those memories interferes with
subsequently reaching any other ones. In any case, I cannot think of any
aspect of consciousness that could operate without making use of short-term
memories, and this suggests that the term 'consciousness' is usually used in
connection with whatever processes brains use for accessing memories of
their recent states.

This raises the question the extent to which such memories might really
exist inside our brains. Clearly there is a problem: if the same neural
network has been used recently for only a single purpose, then it may still
contain substantial information about what it recently did. But if it was
used for several things, then most of those traces will have been
overwritten -- unless some special hardware has been evolved for maintaining
such records. For a modern computer, there is much less of a problem with
this because we can write programs to store such records inside the
machine's 'general purpose memory". Of course, there will be ultimate limits
on the size of such records, but not on the nature of their contents. For
example, most LISP language systems allow the user to specify that all the
activations of an arbitrary set of program-components will have traces
stored recursively. If you specify enough of this before you run your
program, then subsequently you'll be able to find out everything it did --
and even to simulate running it backwards. However, as we've already said,
having such access does not by itself enable the machine to make a good
interpretation of those records. Certainly a certain degree of consciousness
-- in the sense of access to such records -- is necessary for a person (or
machine) to be intelligent. But even a large degree of such 'consciousness'
would not by itself yield intelligence.

So this finally leads us to some really important questions about what are
the uses of consciousness. It seems entirely clear to me that consciousness
has usefulness. It can't be what some philosophers claim: some sort of
useless metaphysical accessory. On the contrary, there are important ways to
exploit short term memories. For example, one has to keep out of loops --
that is, repeating an unsuccessful action many times -- which requires
knowing what already has been done. Also, after one has successfully solved
a difficult problem, one wants to "assign credit" to those actions that
actually helped. This may involve a good deal of analysis -- in effect,
thinking about what you've recently done -- which clearly requires good
records. Furthermore, such evaluations must be done on various scales; did
you waste the last few moments, and why; or did you waste an entire year?
Why do we use the term consciousness only about the shorter term memories?)
On each such scale, you'd better have an adequate array of memories.
Otherwise you cannot intelligently revise your plans, adjust your
strategies, take stock of your resources, and in many other ways maintain
some control over your future. On how many such time-scales do we work, and
how many different mechanisms are involved with each? Because we're living
in the early times of psychology, no one can yet answer such questions.
Clearly it is time to begin to seek constructive ways to study them. To do
this we should prepare ourselves for coping with complexity, because it
seems unlikely that so many different functions can emerge from a single,
completely new principle.

Then what is the alternative. We'll simply have to face the facts that our
many-hundred-organ-ed brain is not a useless luxury. By the time of your
birth the brain contains hundreds of specialized agencies, and by the time
that you're an adult, most of those systems have probably grown through
dozens of stages of development. Now at various times in those first few
yours, some of those systems create the most supremely useful of all
fictions, namely, that the unwritten novel that constitutes your life is
centered on a principle protagonist -- that you conceive of as your
consciousness, like an actual person inside your head! Some sections of [2]
describe in more detail why this illusion is so useful in life; indeed, in
effect, it makes itself true. But the point of all this is to emphasize that
none of those old simplistic concepts from the past -- those spirits, souls,
and essences -- can help us with that modern task, of understanding how all
those different resources, are constructed, operated and managed. Surely
they work to a large extent as a partially cooperative parallel system --
but also, surely, those are largely controlled (much as Dennett suggests in
[3]) by one or several sequentially controlled systems, which in turn are
assembled from smaller parts. The first sentence in my book [2], attributed
to Einstein, is "Everything should be made as simple as possible, but not
simpler." The first step to take toward doing that is to exorcise those
Spirits from Psychology.

[1] The Emperor's New Mind, Roger Penrose

[2] The Society of Mind, Marvin Minsky