How Your Cheese-Powered Baby Trounces AI
An O.G. AI Researcher on How Being Social and Cultural Makes us Intelligent--and Whether Machines Can Ever Learn Empathy
Astute friends and persons of conscience! Today, we discuss AI. Where, in the roomy interval between utopia and extinction, are we headed? What can we expect from this new houseguest, who was not exactly invited but seems to have taken up permanent residence? Computer scientist Melanie Mitchell has been working on AI since Corey Hart wore Sunglasses at Night. She’s known for both her pioneering research into abstraction in AI and her holistic perspective on What This All Means. Mitchell is a professor at the Santa Fe Institute, author of Artificial Intelligence: A Guide for Thinking Humans (and eponymous Substack), and creator of the most popular complexity science class in the world. (You can take it for free!) We spoke about what AI prognosticators get wrong, why having a body is tied to morality, and what a baby can do that machines cannot.
Jess: You’ve said AI is moving really quickly, but there hasn’t been an emphasis on slow thinking. It reminds me of what the computer scientist Donald Knuth said: “Email is great if you want to be on top of things, but my role is to stay on the bottom of things.” Why is slowness important to you?
Melanie: In my field, there are conferences that have deadlines every few months, and it’s always a rush to get your paper done and make sure you're not scooped. So the scholarship in this area is done very quickly—it’s “Does my new version of some large language model do better on this benchmark than everybody else's?” It’s not conducive to deep scientific thinking. What does it mean if it does? What kinds of capabilities do these systems really have? How human-like or unhuman are they? How are they doing what they do? And what is intelligence anyway?
These questions are shoved aside in order to move forward as quickly as possible. And now we have big companies controlling the space, who have commercial incentives to move fast and not think deeply about what their products are doing and what effects they'll have on the world.
So there are layers of incentives: incentives in academia to publish quickly. The market incentives to stay “on top” and crush the competition.
And there’s the national level, where we have to move fast to keep up with China in this AI arms race. That’s been an excuse from big companies: “Regulation is bad because that will make us lose the war with China.” And copyright infringement? Well, if we don't do it, we'll lose the battle with China.
Is that argument wrong?
I think competition with China is often an excuse used to prevent regulation of AI. I don’t find it very convincing.
It can create new drugs for diseases. On the other hand, it's based on stolen work of artists and writers.
Is there a sense in your AI research community of, “Oh, no, what have we done?”
Definitely. There is the genuine sense that maybe we've done something that we regret.
Yoshua Bengio, a big name in this field, feels that a lot of what he's created is super dangerous and he should work against that. Others, like Sam Altman, have compared themselves to Oppenheimer and AI to nuclear weapons; that comparison gives them a caché that's helpful to them in positioning their companies and products as incredibly powerful. Dangerous, but powerful.
And they need to retain all the power because…?
There's this weird fight between proprietary and open source models. Big companies say, “We should ban open source models, because the wrong people will get their hands on AI. We AI gurus understand these models and no one else does, so we should be the ones controlling and regulating them.”
The same dynamic happened earlier in the open source movement. Microsoft had a monopoly on operating systems. The open source community came out with systems like Linux. Microsoft said, “This is dangerous. These systems won't be secure, they’ll be hacked. We know how to prevent hacking.”
It turned out the complete opposite was true—open source was a much more secure way to build software systems. But open source AI models are a threat to big companies’ proprietary models.
Like Encyclopedia Britannica saying, “Wikipedia is too dangerous. It'll be full of errors.”
And people did say that, right? But now Wikipedia is probably more fact-checked than Encyclopedia Britannica.
My background is in physics and poetry, so I have friends in science and technology, and also in the arts and humanities. I've never seen a bigger schism in their attitudes on any topic. Artists have a visceral anger about AI—they don't want to have anything to do with it. Science and tech folks are like, “Welp, it's here. Let's figure out how we can use this.”
It shows the double-edged sword. AI is incredibly useful. It has already had big positive impacts on science—in weather prediction, in protein structure prediction. If it gets protein structure prediction right, it can create new drugs for diseases.
On the other hand, it's stolen. It’s based on stolen work of artists and writers.
Yes. I'm part of the class-action lawsuit against Meta, which illegally stole my book.
Your tech friends say, “They shouldn't have done that, but that ship has sailed. Let's go on with our lives and use AI in the ways that will help us.” Whereas the artists are saying, “It stole everything that I produce that's important. That's my livelihood. We have to be compensated. These things are just immoral.” I see both of those views.
Early on, Open AI and Google had teams to evaluate “Is this ethical? Is this a good thing to do?” But when things got difficult, all those people got fired.
The computer scientist Melvin Conway had this idea that the structure of an engineering team will influence the structure of the product they create.
If artists had been involved in the creation of AI and woven into early research, would AI treat art and intellectual property differently?
Early on, Open AI and Google and other companies had ethics teams to evaluate questions like, “Is this ethical? Is this a good thing to do?” They were proud of that. They put the teams on display.
But when things got difficult, all those people got fired. There were big disagreements between the people who were doing ethics and the people who were pushing the research and products more forcefully. The ethics people were the ones thrown under the bus.
They were welcome as long as they didn't interfere with the business.
It sure seems that way. So I'm wondering if the same thing might've happened if artists were there saying, “Wait a minute, you have to compensate artists if you're going to use their work.” It might have been, “Well, you're out.”
You’ve said that while AI may advance science, it relies on prediction rather than understanding. It may solve a complex problem like curing cancer, but without knowing why.
What do we lose if we gain solutions without understanding?
I think science is not unified in what its goal is. Some scientists want good prediction, some want fundamental understanding. It’s not clear how AI systems are going to help with that.
Although, AI systems are able to identify patterns in data that humans might not be able to see. If they see a pattern that correlates with a phenomenon, maybe then humans can say, “Okay, we never knew there was that pattern, but now we can try to investigate what it’s about.”
Teaming of humans and machines is what's going to give us progress. Machines alone aren’t going to yield a lot of new understanding.
I think this teaming of humans and machines is what's going to give us progress in science, both in prediction and understanding. Machines alone aren’t going to yield a lot of new understanding.
If we don’t understand AI, and AI doesn’t understand itself, doesn’t that turn us all into religious adherents? If it's not important that we understand, AI becomes a faith-based experience.
Yeah, but to be devil's advocate, our relationship with a lot of technologies is faith-based. Here in New Mexico, in court cases, to decide if a person is a flight risk, judges use computational tools that are not that sophisticated as an input for deciding if somebody is granted bail. The tools are proprietary; we don't know why they make the predictions. They have biases that you wouldn't want in such a system. And people trust them.
When people talk about AI risk, the assumption seem to be that machines will inevitably develop the very worst qualities of humans—destruction, subterfuge, manipulation. Why couldn't AI also develop positive traits like empathy?
It’s hard to answer since empathy is a feeling we humans experience, and machines don’t have feelings. No one knows how to give them feelings, since we don’t know how our own feelings work. That’s not to say that giving machines the ability to feel is impossible in practice. It could be argued that we humans are very complex, evolved machines of a certain kind. This is, of course, an old debate— are we basically machines or is some kind of élan vital needed for sentience?
What about other positive traits, like humility or collaboration?
AI intelligence has, for 70 years, been focused on rational thought—the ability to solve problems, do math, play chess.
These assumptions about intelligence hold that the more rational you are, the more intelligent you are. In the game theoretic context, the more intelligent you are, you're looking 20 moves ahead, and the only thing that matters is getting to your goal. Therefore you use subterfuge and all these different negative things to get to your goal. That's how they think about intelligence.
And so when they say “super-intelligent AI,” they're thinking about AI systems that will be monomaniacally trying to achieve some goal by any means because that’s “rational.”
But it's also rational to cooperate.
Sure. But the view of some is that if you give a super-intelligent AI system a goal, then by logic, it’s going to have to achieve some sub-goals. They call these “instrumental sub-goals.” For example, to achieve your goal, you have to continue to exist. So you’ll develop self-preservation. You also probably need as many resources as you can get to make sure that you achieve your goal, like energy or materials.
Humans aren’t sitting above society, they're embedded in it. The assumption for AI is it could be super intelligent, but also be outside of society.
So there’s this whole argument that AI will focus on self-preservation, accumulating resources, and reproducing itself.
Wow, that sounds like a person operating in a highly competitive market-driven environment, like corporations or academia. It seems like an AI researcher is looking in the mirror and and seeing AI.
There's a great article on this by Ted Chiang, a science fiction writer who writes about AI. He says, when AI people think about the risks of AI, they're really thinking about the risks of capitalism.
The idea that AI is going to focus on its own self-preservation, accumulating resources, and reproducing itself also contains an assumption that AI is separate from and in a hierarchical relationship with everything else around it.
That's an assumption. That is not an undisputed truth. But it’s treated as an axiom.
I think so. And it doesn't make sense for any human to do that, right? Humans aren’t sitting above society and culture, they're embedded in it. The assumption for AI is it could be super intelligent, but also be outside of society.
It is a lot built on a house of cards of assumptions. So that's how I think about existential risk arguments. And so I'm not super worried about that.
That’s a relief.
But that's not to say that I'm not worried.
I never use it for writing. Writing's an activity that really helps me think.
Tell me more.
There are a lot of reasons to be worried. These systems can be used to impersonate people, spread disinformation, scam the public. They can be biased in ways that affect civil rights and privacy. There are real reasons to be worried.
In the field right now, there are the “accelerationists” who say, “AI is going to produce a utopia, cure cancer, solve science. If we regulate it, that will stifle innovation.” Then the doomers say, “It's going to kill us all.”
And people in the middle are saying, “No, it’s the things it's doing right now that we should be worrying about.”
You’ve talked about “teaming with” AI. What's your personal rubric for when you use AI, and when you use your own brain and body?
I probably use AI less than a lot of people. I do use it to help me write code-- I'm not that good at writing code. I use it for formatting references. I use chat GPT instead of Google Translate-- it's much better. I never use it for writing.
Why?
I hate its writing style. Writing's an activity that really helps me think. I find writing hard, but it’s good. It’s part of the way I'm understanding things myself.
I also find that the physical act of writing cements ideas more deeply within me. It encodes the information.
Yes. Figuring out how to phrase a sentence or structure something is part of understanding what you're trying to say and how to say it. So I don't use it for writing. The only other thing I use AI for is to try to do my research on it.
Machines need enormous amounts of data. And they still can't do some things a 2-year-old can do.
In the 1980s, if you could have seen what’s happening now in AI, would you have been surprised?
It was unimaginable back then. No one thought that machines could absorb so much knowledge and capacities from just language.
During the Cold War, people wanted to translate between English and Russian, so translation was one of the first AI applications. They built in rules of grammar and linguistic properties of languages and tried to have the systems reason. That was not very successful.
There's this famous dictum in AI, “The Bitter Lesson,” which is that you should never try to “program in” the rules you think will be useful. It always works better for machines to just learn from data or experience. And that’s the new approach.
It's the difference between letting a two-year-old just listen to people having conversations, and sitting down with them and saying, “This is how subject-verb agreement works.”
<Laughs.> Exactly. But there's one interesting difference. How many words are children exposed to up to age 10? Machines are trained on like 5,000 times more. They need so much more training data to get to language.
And they need a lot more energy. You give your child macaroni and cheese and they can talk to you, whereas Microsoft wants to acquire a nuclear power plant to power its AI.
Macaroni and cheese isn't even that high quality.
Right. Children don't need a lot of calories to do what they do. The machines need incredible amounts of electricity.
Why are humans so much more efficient than machines?
People are struggling to figure out the answer. We might have some special, built-in capacities for acquiring language. Also, babies learn differently. Children are actively deciding what they want to learn. They're doing experiments-- dropping Cheerios off their highchair to see what happens physically, how people react. And children have a drive for learning, for understanding.
Machines don't have these. They’re passive. We give them a bunch of text from the internet and digitized books, and they just process all this data. So they need enormous amounts of data. And they still can't do some things a 2-year-old can do.
Some people hypothesize that being embodied in the world-- actively doing stuff to the world-- allows us to learn so efficiently.
It is our social embedding-- our cultural embedding-- that makes us intelligent.
I'm so glad you brought up embodiment. The idea that we could create human-level intelligence without a body strikes me as bizarre. Isn’t the body fundamentally part of our intelligence?
I think about this a lot. It’s certainly very important to our intelligence. This is controversial both in AI and psychology, but I think that our brains were evolved to control our bodies. That's what they're for.
Since its inception, AI has been somewhat dualist in its philosophy—that we can separate intelligence from all that body stuff and just implement the intelligence part in computers.
And this dualistic idea—that the mind and body are separate—is an inheritance of the Enlightenment. Descartes said, “I think, therefore I am” not, “I feel, therefore” or “I'm in a body, therefore I am.”
When you hear people talking about AGI, “artificial general intelligence,” what they mean is logical thinking, not embodied or socially embedded intelligence, which is what we humans really have. But any psychologist will tell you that an individual human alone is not very smart. It is our social embedding-- our cultural embedding-- that makes us intelligent.
Individually, we humans aren’t that different from, say, chimps. But unlike chimps, we rely on culture more that we realize—language, writing, tools, other technologies, the accumulated knowledge and practices that have been handed down to us. We also rely on being embedded in a social world where different people have specialized knowledge. I don’t know how to fix a broken leg but I know how to find a doctor who can.
Thinking as a disembodied brain in a vat is a very different way of thinking than people do.
What is the risk of asking a non-embodied, non-embedded entity for intelligence?
One possibility is that you can't capture robust, trustworthy, morally thoughtful intelligence in a disembodied way. Moral reasoning involves complex concepts. Consider the command “do no harm.” There are situations where “doing harm” is for a greater good—giving a child a vaccination by needle will hurt them temporarily, but we do it for the good of immunization.
Reasoning about moral concepts like harm—or “fairness” or “trust”—requires enormously context sensitivity. I’m not sure how a machine could learn such rich and subtle concepts and reasoning without actively interacting in the world. It might be possible, but I don’t think we know yet.
The End of Bias: A Beginning, about how people become measurably less biased, is out in paperback.
The "AIs' that we use are primarily trained on a corpus of English language texts. I have read that because the training sets are very similar, the responses of various models is rather similar. I wonder how much of our culture becomes evident in the AI. If we train an AI on e.g., only Chinese texts, would such an AI reflect more of a Chinese cultural response? I would argue that as we are embedded in our culture, so must AIs be embedded in their training corpus. Could being more selective in the training set be used to tailor the responses of an AI to a desired cultural model?
I clicked for the "cheese-powered baby" and earned a thought-provoking read. Thank you.