Some of the people building the world's most advanced artificial intelligence systems are suddenly telling us they are worried about where those systems are headed.
Their concern is not that today's AI has become conscious. There is no evidence that it has. Their concern is that increasingly capable AI can reason, plan, act as an agent, pursue objectives and play a growing role in developing the next generation of artificial intelligence. OpenAI chief scientist Jakub Pachocki has even used the term “alien mind” to describe the challenge of understanding increasingly advanced AI.
Anthropic CEO Dario Amodei and other leaders in the field are now arguing that development of the most advanced frontier AI may need to be paced so that safety research and human oversight can keep up. Among their concerns is recursive self-improvement, the possibility that AI becomes increasingly involved in the research and engineering required to build more capable AI.
Listening to this debate reminded me of a much simpler question I began thinking about months ago: How does an infant learn?
A newborn enters the world knowing remarkably little. It does not understand language, recognize danger or know that fire is hot. Yet within a few years, that same child can speak, reason, recognize thousands of people and objects, understand cause and effect, express preferences and begin developing a distinct personality.
No one downloads this knowledge into the child's brain. The child learns by observing, listening, experimenting, imitating and remembering. A face appears repeatedly and becomes familiar. Sounds become associated with people and objects. A child touches something hot, experiences pain and learns not to touch it again. Millions of experiences gradually become knowledge.
Artificial intelligence obviously does not learn exactly as an infant does. A biological brain and an artificial neural network are profoundly different. But the similarities are interesting.
AI also learns through exposure to enormous quantities of information and by discovering patterns and relationships within it. Instead of repeatedly seeing its mother's face, an AI may encounter millions of images and descriptions of faces. Instead of learning language by listening to parents and siblings, it identifies patterns in quantities of human language no individual could encounter in a lifetime.
The mechanics and scale are radically different, but both processes involve turning exposure to information into increasingly sophisticated representations of the world.
There is, however, an enormous difference.
When a child touches a hot stove, it feels pain. Human beings experience hunger, fear, affection, curiosity, satisfaction, embarrassment and grief. Those experiences do more than add information to our brains. They become part of our personal histories and help shape our judgment and personality.
Today's AI does not appear to have anything comparable. It can explain pain without feeling pain. It can write beautifully about grief without grieving. It can describe love without loving anyone.
This is where intelligence and consciousness have to be separated. A machine can become extraordinarily capable of reasoning about human experience without experiencing anything itself. There is no convincing evidence that today's AI possesses consciousness or an internal sense of self.
But what happens as we begin adding some of the other characteristics we associate with intellectual development?
Suppose an AI has persistent memory. It remembers what happened yesterday, last year and ten years ago. Give it cameras, microphones and other sensors that allow it to continuously observe the physical world. Allow it to learn from the consequences of its actions, formulate objectives and determine how to accomplish them. Let it interact continuously with humans and other artificial intelligences.
Then let it operate for twenty years.
Imagine two initially identical AI systems. One spends those twenty years interacting primarily with scientists, mathematicians and engineers. The other spends them immersed among musicians, artists and writers. Their original programming might be identical, but after twenty years their accumulated experiences, memories and knowledge would be radically different.
Would they still be identical?
It seems unlikely.
With human beings, we would say their different experiences helped make them different individuals. With machines, we might resist calling the result personality, but eventually we may have to decide what else to call it.
Creativity raises a similar question.
Shakespeare did not create in an intellectual vacuum. He learned language, read other writers, absorbed history and religion, observed human behavior and lived within a particular society. Beethoven studied music, listened to other composers and spent years mastering his craft.
Human creativity builds upon what came before.
Artificial intelligence is now absorbing human knowledge and creativity on a scale no individual human being could approach. AI can encounter Shakespeare, Homer and Hemingway, study Bach, Mozart and Beethoven, and analyze millions of paintings, scientific discoveries, mathematical proofs and philosophical arguments.
That does not make AI Shakespeare or Beethoven. But the familiar argument that AI cannot truly create because it learns from existing human work overlooks something important: Human beings learn from existing human work too.
The difference may ultimately lie not in whether humans and machines learn from what came before, but in whether there is a conscious individual behind what is created.
That brings us back to today's debate among AI's creators.
The people building frontier AI are not telling us their machines are conscious. Planning is not consciousness. Pursuing an objective is not desire. Remembering information is not the same as possessing an identity. An AI could conceivably display all of these behaviors without experiencing anything internally.
But something important is happening when the vocabulary surrounding AI shifts from tools that answer questions to agents that reason, plan, remember, pursue objectives and increasingly participate in building more capable successors.
The question, therefore, may not be whether AI learns exactly as an infant does. Clearly it does not. Nor is the question whether today's AI possesses a human mind. There is no evidence that it does.
The more interesting question is what happens when learning, memory, reasoning, accumulated experience, autonomy and self-improvement are combined in a single artificial intelligence and allowed to develop continuously.
We know that an infant's accumulation of learning and experience eventually produces an individual human mind. We don't know what increasingly autonomous AI's accumulation of learning, memory and experience will ultimately produce.
But when the people building the world's most advanced artificial intelligence systems begin debating whether those systems are developing too quickly, perhaps the question is no longer quite as far-fetched as it sounds.
