I’m attracted to a particular aesthetic of weathering the storm of societal change to allow fundamentally new things to happen that we couldn’t have imagined before, and of slowly waking up to the fact that most of what we see as foundational, normal, and eternal is actually shockingly contingent, provincial, and recent. There was a time in the not-too-distant past when pizza restaurants and movie theaters didn’t exist, and in 50 years there will be new aspects of the world that feel at least as fundamental to life as pizza restaurants and movie theaters do to me. I associate this vibe with a particular type of minimalism that leaves music, images, and buildings open to being interpreted in wildly new ways as the society around them changes. Some examples are the abstract brutalist monuments to World War 2 in Yugoslavia, which itself no longer exists:

Or specific songs by Brian Eno where he lets a simple, almost silly melody repeat and build until it becomes overwhelming:

Or a lot of New Order songs with such simple melodies and general lyrics that they feel timeless, even though listening to them in 2026 is probably a wildly different experience for me than it was for someone in Manchester in the 1980s:

I like the idea of keeping a part of myself a little detached from the particularities of culture at this exact moment, in the same way these songs and buildings are, so I can learn to let those particularities go as culture changes.

For all these reasons, I’m very attracted to the view that AI will not endanger our survival but will make the world unbelievably weird and new. That’s a world I feel emotionally prepared to handle. I can calmly adjust myself and hold onto the vibes and feelings I experience as most important, which depend so little on the specifics of the here and now.

Subscribe now

I find that I’m also tempted by this intuition when I’m walking around DC, thinking about what could change. When I walk by my local cafe or bookstore, I feel this deep sense that the writing I was reading earlier about AI risk or potential dystopian futures with AI is so abstract that there’s something fundamental about the world it can never touch. The bookstore or cafe feels much more fundamental to what I experience as real, with its ties to my core memories of other cafes and bookstores far away and in the past. It’s like these everyday things flow from some deep, wise center of the world, whereas conversations about AI risk are just surface-level. While I expect the world to change a lot, it feels like this flow from the “real” world will be what changes it. I can’t really describe the feeling better than that, but it’s a very strong and noticeable experience. It’s as if the deep normality of these everyday buildings around me is forming a barricade protecting me from the weirdness of speculative AI risk. I want to be the type of guy who can respond to the subtleties of navigating a bookstore more than I want to be the type of guy who can write and argue about AI risk. Knowing about some specific relation between two obscure used continental philosophy books while sipping a coffee and wearing a tasteful overshirt feels like I’m accessing some deep part of the world that was always there waiting for me to find it, whereas AI risk feels like something shallow and two-dimensional, or like a language game I’ve learned to play with myself.

However, when I step back and ask what’s attracting me to this view, what’s making me feel protected by the everydayness of my specific experience in DC, and what’s making me feel like AI risk is low, it seems pretty obvious that thinking about society-level catastrophe just doesn’t fit with the type of guy I want to be. It feels a lot like I’m looking down at a buffet of options, choosing a bit from effective altruism, a bit from Tyler Cowen-type beliefs about capitalism and change, a bit from the normal techno-optimism that’s defined a lot of my life so far. I put all these vibes on my plate and enjoy the combination of flavors and what they say about my own ideological taste.

I don’t think any of this is an adult way to think about the next 10 years or my relationship to them. All these same impulses would’ve caused me to downplay the threat of the World Wars before either happened, or nuclear risk during the Cold War. I have a strong aversion to thinking seriously about catastrophe. It’s very emotionally intense for me, but also (maybe more importantly) it just doesn’t give off the vibe I want. I suspect a lot of other people have a similar experience.

I’m pretty proud of how I’ve resisted this impulse in the past. In the weeks before COVID hit, I became the local prophet of doom at the school where I was teaching. About a week and a half before we shut down, and before other teachers were giving the same advice (most of them were still laughing off the idea that school would actually close), I started announcing to my class that they should all take their stuff home with them every day, because I expected school to close pretty soon. I wasn’t excited to play that role, but it was pretty easy to see then that COVID was spreading very fast, was gonna be all over the US very soon, and had a reported death rate high enough that it was probably going to cause a lot of things to shut down for a while. I had a record of never crying wolf. I had been pretty cool-headed about other past threats and cataclysms. One day when an anonymous school shooting threat came in, I went into work anyway, knowing the base rates of dying in schools are so low and the false positive rate for threats is so high that being there was still safer than my drive to and from work. I also ended up spending more time than I expected talking students through their unreasonable levels of climate doom, trying to show them that it was possible to be very concerned about climate without believing it would wipe out society in just a few years. But I was able to override my fun, upbeat, cool-headed vibe for COVID because the trends became so obvious that I couldn’t carry them forward without seeing schools close. I was willing to risk crying wolf and break with my normal persona. The buffet option of staying chill was tempting me the whole time, and I had to be more adult, ignore it, and try hard to think about what I actually believed. That turned out to be the right decision!

None of this is to say that our mental aversion to thinking about big cataclysms is any evidence that AI will actually cause one. That’s a separate argument. But in thinking through what you actually believe about AI and its effect on the world, it’s extremely important to avoid this specific move of simply choosing the option that coheres best with your vibe. Instead, you need to look for things to grip onto in the real world. The future of AI might be impossible to fully understand or predict, but we can at least try to think through what our best knowledge of AI, economics, geopolitics, government, and human behavior tells us, and be honest about what they’re actually implying along every step.

This is one reason I’ve been pretty disappointed to see the phrase “I’m not dooming” about AI come to mean “I’m not giving in to negative emotions.” Beliefs about AI shouldn’t come from your emotional state either way. Most people I know who worry about AI catastrophes (myself included) are not predisposed to “doom” about other things. If anything, we tend in a techno-optimist direction otherwise. This general implication that “doomers” believe what they do because they’re neurotic just doesn’t fit my own experience. I constantly have to drag my own mind kicking and screaming to think through AI risk arguments, because they just don’t match my emotional temperament or vibe at all. And yet many of the arguments are so straightforward and compelling to me that I’d easily be classified as a “doomer” (in the sense that I think there’s a real chance AI destroys civilization) despite having pretty low neuroticism. Of course I could be getting those wrong, but my and your emotional state don’t really help in figuring that out. As a claim about your emotional state, “I’m not dooming” is pretty uninteresting, because your emotional state gives you basically no access to the actual future of AI.

Many people understandably think of a lot of talk about AI risk as a kind of social game, and their own belief as something like a specific style of clothes to wear in this game. That might have made more sense 10 years ago, but the trends in near-term AI are looking uncertain and potentially wild enough that it’s time to start thinking about it at the level of seriousness you would have wanted in the early days of the Industrial Revolution or the Cold War. Simply testing out different beliefs about AI as if you’re sampling buffet options isn’t a sufficiently adult way of thinking about the next 10 years of your own life.

Subscribe now

Like the last post, this one’s long. This series is me imagining I had a few hours with a smart friend to get across what I actually think about AI and philosophy of mind, and trying to do my very best.

To get to my full argument, I first need to argue that it’s possible in principle for computer programs to think and understand, and only then can I claim that specific AI programs can. This post argues for that first claim, which is pretty contentious on its own, and the next post will argue for the second.

It’d help to have Part 1 fresh in your mind before going on, or at least the section on introspection, the argument that physics is causally closed (and what that means for phenomenal consciousness), and the concluding summary.

I’m not claiming in this post that current AI models think (that’s Part 3). I’m not claiming the brain is a computer, which I’ll explain below. And I’m not claiming AI is conscious. I don’t think there’s anything it’s like to be a current AI model. In Part 1 I argued that the phenomenal consciousness question doesn’t affect whether AI models can think anyway.

Here’s how my argument in this post will go. I argue for each of these premises, and the conclusion follows:

• Premise 1: Things that think and understand are reducible to physical processes. There’s no extra nonphysical ingredient.

• Premise 2: What makes a physical system think is how it processes information, and what it’s made of only matters insofar as it can do that processing.

• Premise 3: A computer can simulate the brain’s information processing to whatever level of detail matters.

• Premise 4: A detailed enough simulation of an information process is that process, running in a new substrate.

• Conclusion: A computer process can, in principle, think and understand.

Most people who reject the conclusion reject premise 2 or premise 4, so those are what the majority of the post will focus on.

I won’t claim that AI models are or are meant to be simulations of human brains. This argument is only to show that because a simulation of a human brain could think and understand, computer processes can in principle think and understand. Once I’ve shown it’s possible, the burden of proof will still be on me in Part 3 to show that current AI models are sepcific examples of computer programs that can think and understand.

Part 1 took qualities we tend to assume are fundamental to minds and tried to demystify them a bit to show that they might be available to machines. I started high up in our conception of what humans can do and slowly worked us down to a more natural and nuanced picture of the human mind. This and the following post will build up from the ground instead. I’ll start with the basic building blocks of mind in physical reality and work my way up from physical systems (premise 1), to non-carbon-based material (premise 2), to computer programs specifically (premises 3 and 4). Part 3 will then argue that the internal workings of AI models specifically meet the definition of thought.

Why computers can in principle produce thinking and understanding

What are minds made of?

Can physical objects think?

The first and simplest step in figuring out whether computers can think is to ask whether merely physical objects can, or whether thinking requires something nonphysical, something more than particles, forces, fields, and the ways they relate to each other. I discussed this a lot in Part 1 but will revisit the core points.

I see 3 possible answers:

1. Everything about the mind is physical, including thought and understanding.

2. Some parts of the mind aren’t physical, but thought and understanding aren’t among them, and don’t depend on anything nonphysical. Maybe our phenomenal consciousness (what it’s like to be us) is nonphysical, but our thought and understanding are a kind of physical information processing.

3. Thought and understanding themselves depend on something nonphysical. They happen in part outside of the physical world.

Number 3 would mean we probably can’t build machines that really think and understand, while 1 and 2 leave the possibility open. I argued in Part 1 that while phenomenal consciousness is still a deep mystery, real thought and understanding don’t seem to depend on it. The only kind of consciousness they need is what philosophers call “access consciousness,” which is a specific, elaborate kind of information processing. So for the purposes of this series, I’m going to go on assuming #1 or #2 is true, and both say that it’s possible in principle to build a purely physical object that thinks and understands. Under either view, human brains are proof that this kind of object can exist.

I also see lots of reasons to think that physics is causally closed. As I explained in Part 1, it seems suspicious to assume that the one place in the world where physics is regularly violated is inside our own heads. This would also seem to break basic important physical rules like the conservation of energy, that would turn the world topsy turvy if they were regularly broken. And it seems weird that a purely physical, unguided process like evolution by natural selection could have stumbled on some way of interacting with nonphysical stuff. This makes 1 and 2 seem especially likely to me. A lot of people I meet who believe 3 believe it for religious reasons, and that’s fine, but if that’s your main reason for believing that AI can’t think, it’s important to say so directly instead of making some unrelated argument about the science of physical brains and computers. Some also believe #3 for scientific and philosophical reasons, and that’s also fine, but I think the burden of proof is on them to demonstrate why they do.

Under the view that the mind is reducible to physics, our thoughts and understanding are high-level patterns that emerge in our physical brains. Here a mind is kind of like the national debt, which isn’t a physical entity itself but is ultimately the result of huge numbers of specific tendencies and patterns in the behavior of purely physical systems.

So I think the answer is either the first or the second, and either way, things that think and understand are reducible to physical processes. There’s no extra nonphysical ingredient. That’s premise 1 of my full argument. The fact that minds happen in physical objects has a lot of additional implications for philosophy of mind.

Implications of minds existing in purely physical objects

In classical Newtonian physics, everything that happens is predetermined by the initial state of the physical system you’re in. If you had absolutely perfect information about the beginning of the universe and infinite time to calculate everything, you could in principle perfectly predict everything that would happen, down to the fact that you’d be reading this blog post right now. If you could rewind a Newtonian universe, the exact same thing would happen when you played it back.

We now think this probably isn’t true. On the standard reading of quantum mechanics, events at the quantum level are fundamentally probabilistic. They aren’t determined in advance, but they can be described and estimated using probabilities. If you rewound the universe and started it over, different things could happen. Some interpretations of quantum mechanics (like my favorite, many-worlds) circle back to a kind of determinism, but even here they involve what looks like randomness from the observer’s point of view. Either way, physical reality reduces to strict laws plus randomness.

Everything we know of in physics either follows strict deterministic laws or is random in the ways quantum mechanics describes. This has an important implication for brains and thought, assuming that they are ultimately physical: any brain activity or mental process is either predetermined by physics, random, or some combination of the two. If your requirement for real thinking and understanding is that it can’t fundamentally rely on either purely mechanistic systems or randomness, then either humans have never thought or understood anything, or human thought transcends physics. Both seem wrong to me, so it looks like real thinking needs to be able to be broken down into purely mechanistic systems or include pure randomness, and there’s no secret third thing involved.

This gives us a quick way to check a lot of arguments that AI can’t think, which I’ll call the brain test. Take the argument and apply its reasoning to the human brain instead. If the reasoning also applies to the human brain, the argument proves too much, because it would also show that you can’t think. I’ll come back to this simple test a lot.

Another way of saying this, which I described in Part 1, is that any system that understands is ultimately made up of individual parts that don’t:

Unless thought and understanding are supernatural and happen outside of physics, any system that understands or thinks is ultimately made up of fundamental particles and forces. These on their own don’t understand or think. Thinking is an emergent property of arranging them in specific ways to perform specific functions. At a higher scale, individual neurons also don’t “think” or “understand.” They are building blocks for a specific system that thinks and understands. All that human neurons do is receive basic chemical signals, change their electrical activity, and release some chemicals. Any system that understands will be made up of parts that “only” do simple things. That’s a basic part of living in physical reality.

…

The role of access consciousness in thought, like anything else in the world, also needs to be completely explained by individual processes that are not conscious. If thinking requires some higher-level awareness, that awareness also has to be explainable in the language of physics (again, unless you believe it happens in a separate nonphysical world that interacts with ours). Dennett once observed (in his book Brainstorms) that explaining an intelligent system means breaking it into smaller, dumber systems, then breaking those into even dumber ones, until you get down to parts so simple that they can be “replaced by a machine.” I think this is the only way minds can ever be explained.

Information theory

My argument later on will be that thought and understanding are fundamentally both specific ways of processing information, so it’s important to explain how information fits into a purely physical view of the mind. There’s a field of math and physics called information theory (started by Claude Shannon, who the AI Claude is widely thought to be named after) that gives us a precise way to measure information. Information is anything that reduces uncertainty, and it’s measured in units called “bits,” where 1 bit is the answer to a yes or no question when both answers are equally likely. If I have a number in my head between 1 and 8, it takes 3 bits of information to pin it down, because you can ask “Is it in the top half of the range?” and then “Is it in the top half of the remaining range?” and then repeat that second question one more time. If I thought of 6, the answers would be yes, no, yes. Each bit doubles the number of possibilities you can tell apart. One bit can tell you if a number is 1 or 2, two bits can tell you which of 1, 2, 3, or 4 it is, three can tell you which number from 1 to 8 it is, and so on. This is why the game 20 Questions can get you to one specific thing, person, or place out of all the options, because 20 bits can narrow things down to 1 option out of about a million, 2²⁰.

Under this definition, to say that a physical system “carries information” means that its state changes reliably with the state of the other thing. Tree rings carry information about a tree’s age, for example, even if a human never actually counts them. To say a system “processes” information means that its information-carrying states cause each other to change in ways related to the information that changes and shapes what the system does. Sometimes people use “information processing” to mean the very specific way that a computer manages its own information flows. Computers work very differently from the human brain, so people will sometimes say that brains “don’t process information.” This is true in the limited sense that they don’t behave exactly like computers (your memories for example aren’t perfectly preserved in clear places for retrieval), but false in the broader sense that information theory uses the term. This semantic overlap leads to a lot of problems I’ll discuss more below.

Each bit needs to be stored in or carried by something physical (and since nothing that carries a signal can move faster than light, information can’t either), but the information itself is independent of the specific thing it’s stored in. It can be transferred from one thing to another. For example, the words on your screen right now started off as information in my brain, which became patterns in how I typed the words into my computer, which became both words on my screen and electric charges in my laptop’s memory. When I added them to my blog, the information traveled as radio waves to my router, then mostly as light in fiber optic cables to a data center somewhere, where it was stored again as electric charges in the chips there. Then it reached your device the same way, came out of your screen as light that you read as words when it hit your eyes, and now it’s being stored in your brain. This was the same information all along. It could be carried by patterns in very different physical stuff, whether brain tissue or light or electricity or the movement of my hands on my keyboard. But this information isn’t some special nonphysical substance. It’s just a very complex, specific pattern in physical material that influences patterns in other physical material, the same way an ocean wave is a pattern in water that shapes the water ahead of it without being anything magical over and above the water. A wave can even cross an ocean while each particle of water just bobs in place.

Does an object need to be made of a specific material to think?

Here I’ll argue for premise 2: what makes a physical system think is how it processes information, and what it’s made of only matters insofar as it can do that processing. This is often the most controversial part of the argument.

A belief that’s very popular among everyday people is that thinking depends on biological material. A lot of people believe that there’s something so fundamentally special about the material the brain is made of that a similar object made of anything else couldn’t “really” think or understand. At best it would just be an imitation. Obviously, to make my case I need to convince you that thinking and understanding can happen in non-biological material like silicon.

The word “substrate” will be useful here. It comes from a Latin word meaning “spread underneath.” In philosophy of mind, it means the physical stuff a process happens in. The substrate my computer programs run on is silicon, and the substrate my mental processes run on is my brain tissue. “Substrate dependence” is the view that mental events like thought or understanding can only happen in very specific substrates, and (on most versions of the view) silicon isn’t one of them.

There are two possible reasons why mental activity might depend on specific substrates:

1. Only specific substrates can play the causal, functional roles needed for specific mental processes. For the same reason that you can’t make a calculator out of a puddle of water, you can’t make something that thinks and understands out of just helium. A lot of substances don’t allow the kinds of complex organization and structures that minds need. Maybe carbon-based biological substances are the only ones that can form the types of causal structures that thought and understanding require.

2. Specific substrates have some inherent quality that lets things made of them think or understand, over and above the causal roles they play in the systems they’re in. Maybe only biological brains can really understand anything, because the material they’re made of is somehow tied to what understanding is, beyond any functional role it plays. This view says that even if you could perfectly remake every last process happening in the brain using a different material, the artificial brain would only be imitating understanding and thinking.

To see the difference between these two views, we can imagine (following a famous thought experiment from David Chalmers called “fading qualia”) that scientists have made a surprising discovery: they can replicate literally every process that happens in your brain using silicon-based material rather than carbon-based material. There is no causal difference in the brain’s behavior when they replace a carbon-based neuron with a silicon-based one.

Suppose that they replaced a single neuron in your brain with one of these silicon neurons. What would happen to your consciousness and thinking? Would you notice? What would happen if they slowly, over the course of months, replaced more and more of the neurons in your head with these silicon-based neurons?

If you hold belief #1, your mind would remain intact, because these artificial neurons can (for the sake of the thought experiment) in fact play the same functional role that carbon-based neurons play. There would be no change in your perception, thinking, or understanding. The only thing about the carbon-based neurons that mattered for your mind was the specific role they played in your overall brain processes. Replacing them with these new neurons would be like replacing individual transistors in your computer with new ones that work the same way. It could still run all the same programs.

If you hold belief #2, replacing your biological carbon-based neurons with silicon neurons would slowly remove key parts of your mental activity. Maybe your field of vision would fade or shrink. Depending on what you believe is reliant on carbon, by the end you either wouldn’t be conscious, or wouldn’t be able to really think and understand, or both. Your outward behavior would stay exactly the same the whole time, because the cause-and-effect relationships that made up your brain would all still be running, but there would be no real thought, understanding, or consciousness behind it. This is almost exactly one of the scenarios John Searle described in his 1992 book “The Rediscovery of the Mind,” where, as your brain is replaced with silicon, you internally want to cry out that you’re going blind while your voice, completely out of your control and now run by the silicon neurons, keeps describing what’s in front of you.

I’ll call belief #2 “the carbon view,” and I think the thought experiment itself shows that it’s wrong. I’ll also argue later that belief #1 is correct in some sense (you can’t make a mind out of helium because it couldn’t maintain the right cause-and-effect relationships), but that silicon (and probably a lot of other materials) checks all the boxes for the specific causal relationships and functions we’d need for real thought and understanding.

What would actually happen in this thought experiment?

I want to start to poke at the carbon view, starting with an inconsistency in its answer to the thought experiment above.

Let’s imagine that 10% of your neurons have been replaced with these silicon neurons. Which of these would your field of vision look like?

If your consciousness were slowly being taken away, would you notice? Would you react? Remember that these new neurons have the exact same cause-and-effect structure as your old ones. With either kind of neuron, if your brain received information that your field of vision looked strange, it would cause you to say “Hey, there’s a bit of my vision missing.” But because the new neurons play the exact same causal role as the old ones, you would do and say exactly what you would have done and said with your old brain. If your old neurons wouldn’t have led you to say “Hey, my field of vision looks weird,” your new neurons won’t either.

So if your field of vision really were dissolving, you’d never say anything about it, and what you said about your vision would have come completely apart from what you actually see. If the silicon neurons cause you to behave in exactly the same way, this strong observation of your consciousness changing should trigger the same behavior the old neurons did, but for some reason they wouldn’t here. It seems like the only way for this to work is for you to somehow not notice your own phenomenal consciousness changing. You’d be completely and permanently wrong about your own experience while your brain went on working exactly as before. This seems to imply some kind of epiphenomenalism I talked about in Part 1, where your consciousness helplessly observes your life but never has any causal influence on it.

Searle accepts that this could happen, but I find it much harder to believe than the alternative, that your field of vision isn’t dissolving at all. (This is basically the philosopher David Chalmers’ “fading qualia” argument). Philosophers really do disagree about the experience part. For example, if experience turned out to be nonphysical (option 2 from earlier), physics being causally closed would mean it couldn’t affect what you say anyway, which is one situation where Searle’s scenario hangs together. But the case is much stronger for thought and understanding. The new neurons do exactly the same things with information as the old ones, and in Part 1 I argued that’s what thinking and understanding come down to, so a brain made of this kind of silicon could think and understand. What matters for these new neurons is the role they play in your broader mental system, not what they’re made of. If they play the same role, they’re the same for your mental activity. I see this as a strong argument in favor of belief #1.

Another way to put this is as a dilemma. Either the special ingredient in carbon makes a difference to what the brain does, or it doesn’t. If it does, then a silicon brain that does everything the carbon brain does has something playing the same role. If it doesn’t, it can’t be what thinking is, because thinking makes a difference to what you do, and nobody could ever tell who had it, including you.

The carbon view as I’ve described it is mostly a folk view. The philosophers and scientists who take substrate seriously today (like Peter Godfrey-Smith, Anil Seth, and Ned Block) don’t think carbon atoms are special in themselves. They believe that the functional biological processes like metabolism might matter for consciousness specifically. The debate among experts seems to be mostly about phenomenal consciousness, and they are often otherwise open to machines thinking and understanding language.

Even if you find the replacement argument convincing, the carbon view can still feel obviously right, because every mind we’ve ever seen has been made of carbon. The next few sections explain where that feeling comes from, and why it doesn’t tell us anything about whether other materials can think. If you already don’t believe the carbon view, you can skip ahead to “Some takeaways from neuroscience.”

Is carbon required for thought?

Evolution and carbon

A lot of people believe the carbon view because of a simple observation: every mind we’ve ever seen is carbon-based, with no exceptions at all. Anything that we’ve ever called a mind, whether it’s in a human or crow or octopus, was in biological matter. This seems like strong evidence!

However, there seems to be a much more obvious reason why minds have only appeared in carbon-based biological matter.

Minds need an enormous amount of very specific, organized complexity. We only know of two processes in the universe that can build very complex things that work:

• Evolution by natural selection

• Deliberate design by people

Nothing else we know of can create anything as complex as human and animal brains. And for almost all of Earth’s history, only the first one existed. On Earth, as far as we know, natural evolution has only ever acted on carbon-based biological matter, so only carbon-based material could be shaped into the level of complexity required for minds. Thus, the only things we see that have minds are carbon-based.

Why has natural evolution only acted on carbon-based material?

Well, stepping back, evolution isn’t a force in the world with goals. It’s just an emergent pattern that shows up on its own, because things that survive and replicate more end up producing more things like them. We can expect the pattern of evolution by natural selection to emerge whenever objects in the world have these three properties:

1. They can replicate themselves.

2. They carry information that gets passed on to their copies, and that information can be modified in the act of replication, leading to variation among the copies. The copying has to be mostly accurate, or the information would get scrambled over the generations, but it can’t be perfectly accurate, or nothing new would ever appear.

3. Some of that change affects how good the objects are at replicating in their specific environment, so some versions end up making more copies of themselves than others, spreading their variants more.

Over time, this process can stumble on mutations that add more and more complexity to the species they happen in. With enough time, this creates features that look like they were the product of design, like eyes, wings, immune systems, or minds, even though no designer was involved. For almost all of Earth’s 4.5-billion-year history, this was the only possible source of the level of complex organization minds require.

The basic dynamics of evolution don’t depend on any specific substance. In a famous 2003 study using a digital evolution program called Avida, scientists made self-copying computer programs that evolved to do complex logical operations by building on simpler abilities they had evolved earlier, in the same way complex features emerge in living things over many generations. Evolution can act on anything that checks all three boxes I listed above.

So why has natural evolution on Earth only acted on carbon-based material? It turns out that it’s mostly because carbon is more useful for the kinds of replicators evolution needed to get started on Earth specifically, and is a good, stable material for more complex structures.

The main reason carbon’s so useful and versatile as a building block for replicators is that it has four outer electrons (valence electrons) available to form bonds with other atoms. As it happens, silicon is the very next element with 4 valence electrons on the periodic table.

This means silicon can form some of the same kinds of molecules as carbon. Silane (SiH₄), for example, is silicon’s version of methane (CH₄), but it has some very different properties (it bursts into flame on its own when it touches air). Silicon is much, much more abundant on Earth than carbon. By the most commonly cited estimate, there are around 600 silicon atoms for every carbon atom in Earth’s crust.

If silicon is so much more abundant than carbon, and can form similar molecules and bonds, why were the first replicators that kicked off the process of evolution carbon-based and not silicon-based?

One reason is that when a carbon atom bonds to another carbon atom, that bond is about 50% stronger than the bond between two silicon atoms. This means silicon is much worse at forming the long chains and rings that are required for larger and more complex structures. Chains of carbon atoms are also very stable in water, whereas silicon chains break apart much more easily in both water and oxygen. That matters a lot because the first replicators (which arose at least 3.5 billion years ago) almost certainly formed in water (the earth was mostly covered in it). The silicon versions of carbon’s building-block molecules (like the silane mentioned above) also tend to be much less stable in general.

Silicon also bonds much more strongly to oxygen than it does to other silicon atoms, so in nature it almost always ends up bonded to oxygen. Almost all the silicon in Earth’s crust is locked up in rock as a result. Silicon dioxide (one silicon atom for every two oxygen atoms) is what we know as quartz, the main ingredient in most sand. If silicon-based animals breathed out silicon dioxide like we breathe out carbon dioxide, they’d breathe out sand.

So it seems like the main reason life is carbon-based is that carbon forms strong bonds with other carbon atoms, those bonds hold up in water, and the first replicators almost certainly arose in water. These facts are interesting, but on their own they don’t tell us anything about whether there’s something unique about carbon-based molecules that makes them important for thinking and understanding. What they tell us is that carbon was the basis of the specific molecules that kicked off the evolution of all the complex life we see today. As far as we know, that happened exactly once. Any new replicators that showed up now would almost certainly be eaten or outcompeted by the life that already exists.

So what evidence do we get from the fact that all minds we’ve ever seen are in carbon-based biological brains? All it tells us is that the universal common ancestor of all life (a single-celled organism that lived more than 3.5 billion years ago, roughly 3 billion years or more before the first simple brains evolved) was made of carbon-based stuff, and it was successful at replicating itself and met the three criteria needed to run the pattern of evolution by natural selection, which was the only process on Earth that could produce the level of complexity minds would need much much later on. Until people started building flying machines, the only objects in the world that could fly were also biological and carbon-based, because again, living things were the only things complex enough to fly. This told us nothing about whether there was something inherent to carbon-based biological material that made it necessary for flight. Like flight, thought and understanding can only happen once some process has organized incredibly complex, specific systems. For billions of years the only process that could do that was evolution, and if I’m right, in the last year or so human design has become a second way to create key high level aspects of minds.

If carbon-based biological matter were required for thought, over and above its ability to play the correct functional roles in systems, this would imply a lot of other strange things about the world. For one, on any planet where evolution acted on another type of replicator (maybe one where the environment was more hospitable to silicon-based life), evolution could never produce thinking animals.

This would be a massive missing tool that would’ve been useful to those silicon-based animals. Thinking has obviously been incredibly useful to animals on Earth, to the point that complex thought seems to have evolved independently multiple times, in lineages that split off from each other before any of them had complex brains. For example, humans and other primates, crows, and octopuses can all solve complex problems that require step-by-step planning, and their evolutionary lines all split from each other long ago.

Octopuses and humans last shared a common ancestor about 600 million years ago, long before either lineage had much of a brain. Crows and humans split off about 320 million years ago, when brains were still small and reptile-like. Our shared ancestor looked something like a small lizard, and birds and mammals each separately went on to evolve brains with about 20 times as many neurons as reptiles of the same body size. This means that on at least three separate occasions, relatively simple brains independently evolved to the point that they could solve complex problems like when an octopus carries coconut shell halves across the sea floor to use as a shelter later, and when a crow pulls up a short stick dangling from a string, uses it to fish a longer stick out of a box, and then uses that to get food out of a hole. The wild differences in the look of the brains of the three species are a result of approaching the same broad ability from wildly different starting points:

It is crazy that intelligence has independently evolved this often.

Notice as an aside that when we say a crow thinks, we’re not saying it has the same structure as a human brain. Crow neurons are arranged very differently from humans’. Most of an octopus’s neurons aren’t even in its brain, they’re in its arms. No one concludes from these deep structural differences that octopuses and crows can’t think or only imitate problem solving. We determine that they think because of what their brains do, not whether they have the exact same structure as human brains. We’ll come back to that later.

But according to the carbon view, this ability, which has been useful enough to animals on Earth that it evolved multiple times, could never evolve on that silicon-based planet. Even though silicon could play all the required physical roles in the animal’s broader system, to the point that animals would behave exactly as if they were thinking, it would somehow “not count” as thinking because it wasn’t carbon.

This view makes the history of life look like a bizarre miracle. The first real brains didn’t evolve until a little over 500 million years ago. For more than 80% of life’s existence, there were no brains. What are the odds that the first replicators happened to be built around the one element that, for some separate reason, is the only one that can support real thought and understanding, or consciousness?

Why would this quality of “necessary for thought” even belong to a specific element? Or one that bonds so easily with other elements? Could there be a universe where only neon could produce real thought, but because neon is a noble gas and doesn’t form stable molecules, no thinking things ever evolved anywhere? The universe would be permanently full of this hidden capacity for thought that nothing could ever actually tap into. Does that concept even make sense? It sounds incoherent to me, which makes it hard to believe that any one element has some inherent property that makes brains made out of it “do real thought” while systems made of other stuff that can perform all the same physical functions don’t. Did we just get incredibly lucky that the thought-element happened to be perfect for replicators?

The carbon view looks pretty strange in the context of evolution. But evolution also gives us clues to why we might hold it so strongly.

To indulge in armchair evolutionary psychology for a moment, it seems like any human in the ancestral environment who would treat rocks or other material as potentially other entities like them, entities with complex mental properties, would probably be at a pretty bad disadvantage. They would waste a lot of resources either trying to cooperate with or fight them. It seems like most animals that can perform complex thought have a pretty strong inbuilt sense of what’s a living thing (to either avoid or fight or eat or cooperate with) vs what’s a nonliving physical object. This sense obviously doesn’t directly track carbon, and can misfire. It looks for things that biological life has, like faces or independent motion, which is also why humans often over-attribute minds to things that do either. This inbuilt sense would’ve been shaped by what helped or hurt animals in survival and reproduction, and in a world where carbon-based life was the only stuff that could develop the kinds of complexity that matter for whether animals react to them as other agents in the world, the strong intuition that only things that look like carbon-based life can have minds would have been very useful, regardless of whether it was true about the universe more broadly. There’s no clear causal path from whether other stuff can perform mental activity in principle to the intuitions that humans in the ancestral environment developed. I suspect a lot of our intuitions about philosophy of mind come from something like this evolutionary pressure to not consider nonliving matter as possibly possessing any complex mental behavior, and the thing that created that evolutionary pressure was ultimately the simple fact that carbon is an especially useful base for replicators, which doesn’t actually relate to whether other matter could in principle perform mental activities if arranged in the correct way at all.

Carbon is itself just a specific arrangement of smaller parts

Another problem for the carbon view is that atoms aren’t the most basic physical things reality is built from. They’re made of protons and neutrons and electrons, and protons and neutrons are themselves made of quarks. A normal carbon atom has 6 protons, neutrons, and electrons, and a normal silicon atom has 14 of each. They’re both made of the exact same stuff. There’s just more of it in the silicon atom.

Everything that makes a carbon atom behave differently from a silicon atom is just an emergent pattern resulting from how many parts there are and how they’re arranged. Both have the same number of outer electrons, which is why they can form similar types of molecules. Silicon forms weaker bonds with other silicon atoms mostly because it has one more layer of electrons below its outer electrons than carbon does, which makes it a bigger atom. Every difference in their chemistry comes from differences in how the same underlying particles behave in each.

This, even more than evolution, is what makes the carbon view look so strange to me. It doesn’t make much sense to say that carbon has some quality over and above the role it plays in larger systems, because carbon itself is just the pattern emerging from the individual roles being played by simpler interacting parts, the same parts that make up every other element. If you arrange the simpler parts differently, they do different things, which we call other elements. Carbon is made up entirely of parts that aren’t carbon. How could a mere role that simpler parts play somehow have a fundamental property beyond the role it plays in a broader system?

About 1% of the carbon atoms in your brain are carbon-13, which has an extra neutron. It chemically behaves almost exactly the same as normal carbon, and your body uses it interchangeably. Does carbon-13 also have the same special thinking property as normal carbon? If it does, it’s a little suspicious that the only other type of thing that has the thinking property also plays the exact same functional role in the broader system. Adding a neutron is fine, but a proton is bad? A neutron is itself made of one up-quark and two down-quarks, and a proton is the reverse. Is adding one more up-quark and taking away a down-quark the thing that thinking is so tied to? If the quality of “real thought” disappears with the extra neutron (which barely changes anything the atom does), then about 1% of the carbon atoms in your brain, scattered randomly throughout it, aren’t being used to really think. Again, seems bizarre. And if adding just 1 neutron doesn’t change the picture, why does adding one proton or electron change it so completely, if not for their effect on how the arrangement behaves in a larger system?

If thinking needed something from carbon atoms over and above what carbon does, it’s difficult to see where that would come from. Everything we know about carbon is completely accounted for by what its smaller parts are doing, and it shares those smaller parts with silicon, just with different amounts and arrangements. Carbon itself is just a specific functional arrangement of smaller parts, so if thinking depends on carbon at all, it depends on a particular arrangement of more fundamental particles, the same particles every other element is made of. So thought still ends up as a specific functional process of matter, and only relies on the type of matter if it can perform that function, partly because every type of matter more complex than quarks or electrons is itself just a function of an arrangement of smaller bits and the roles they play in the system. This circles us back to belief #1: thinking depends on a specific substrate only in the sense that the substrate can or cannot perform the specific causal roles required for thought.

So now we need to figure out if silicon-based structures can play roles that are necessary and sufficient for real thought and understanding. To figure that out we need to first look at neuroscience and then computer science.

Some takeaways from neuroscience

Neuroscience is still deeply uncertain about what happens in our brains when we think and understand. However, we can draw some important lessons from it about what’s happening in brains and whether silicon computers could also do it. These are the key takeaways from the field relevant for my argument:

1. Thinking and understanding happen in the brain through neurons passing information to each other.

2. What matters about those signals is the information they carry, rather than the stuff carrying it.

3. The brain’s information processing can be described with ordinary classical physics, and doesn’t seem to rely on special quantum effects.

These support my core premises 2 and 3:

2. What makes a physical system think is how it processes information, and what it’s made of only matters insofar as it can do that processing.

3. A computer can simulate the brain’s information processing to whatever level of detail matters.

Thinking and understanding each happen in the brain through neurons passing information to each other.

A cornerstone of modern neuroscience is the “neuron doctrine,” the idea that the brain is made up of separate cells (neurons) that each receive signals from other neurons (often thousands of them) at contact points called synapses. Some signals make it more likely that the neuron will fire a brief electrical pulse, and others make it less likely. If the combined signals cross a threshold, the neuron fires, sending a pulse down a long fiber that releases chemicals onto the next neurons, pushing them toward or away from firing, and so on. Your brain has around 86 billion neurons and something like a hundred trillion synapses.

The fact that the brain can be broken up into separate units that each take in inputs and produce outputs means that the entire brain can be described in terms of what its units do and how they’re connected to each other. The true way the brain works is messier, with additional cells and chemicals and neurons wired directly to each other, but ultimately all of it is still physical signaling that can be described the same way.

What matters about those signals is the information they carry, rather than the stuff carrying it.

It appears that whatever makes a pattern of activity in your neurons a thought or perception or memory is the information it carries and what the rest of the brain does with it, rather than the particular physical stuff carrying the information. In the 1920s, the physiologist Edgar Adrian found that individual signals in single nerve fibers are all basically alike. A stronger stimulus (like a harder touch or a brighter light) doesn’t lead to larger pulses (or “spikes”). It just makes them happen more often. So what makes some of these signals sights and others sounds is which neurons they reach and what those neurons do with them.

There are already examples of information being processed by silicon before it reaches the brain, the most famous being cochlear implants, which do the job of the cells in the inner ear that normally turn sound into nerve signals, using a computer chip that sends electrical pulses into the auditory nerve. Many implant users can hold phone conversations without lip reading, so the brain is clearly able to build understanding out of information that silicon has processed.

In 2018, researchers worked with eight epilepsy patients who already had electrodes in their brains for medical reasons. They recorded activity in the hippocampus (a part of the brain that’s key for forming memories) and built a mathematical model of the activity patterns that went along with remembering things correctly. They used a computer to stimulate the same patterns into the patients’ brains during later memory tests, and their scores improved by about 35%. A computer model of part of the brain’s information processing was helping with a cognitive job inside a human brain! This seems a lot like the thing the brain was doing was manipulating information that a computer was able to also manipulate in the same way.

For the most part, neuroscientists connect thought and understanding to how information flows through the brain, rather than to which material is carrying it.

The brain’s information processing can be described with ordinary classical physics. It doesn’t seem to rely on special quantum effects.

Most neuroscientists and physicists think the brain doesn’t make use of quantum effects like superposition, and that the way it processes information can be fully described with classical physics. (Quantum mechanics underlies all chemistry, including the brain’s, but that’s different from the brain using quantum effects to process information.) They think this for two reasons:

1. Classical models of neurons work really well. Every successful simulation of neurons treats them as classical, mechanistic systems.

2. Quantum states are incredibly fragile, and the brain is warm, wet, and very crowded with jostling molecules. Max Tegmark estimated that quantum states in the brain would fall apart within 10^-20 to 10^-13 seconds. Neuron activity itself takes thousandths to tenths of a second, at minimum 10 billion times longer than the quantum effects happen.

Two things about classical physics become really important here:

1. Classical physics is mechanistic. The brain is way too complicated for anyone to actually calculate what it’ll do, but everything that happens in the brain, as a system subject to classical physics, is determined entirely by its starting state, its inputs, and the laws of physics. Some randomness exists in things like ion channels that randomly snap open and shut, or synapses failing to pass a signal on. If you rewound the universe and replayed it, some of those random events might come out differently, but otherwise the brain would give the exact same response to the exact same input if it started in the exact same initial condition.

2. Every classical physics situation in the everyday world can be modeled with differential equations (equations that describe how things change from moment to moment). In the brain, these can be used to describe how quantities like voltage and chemical concentrations change smoothly over time.

So the brain’s processes, including whatever thought and understanding are, are reducible to classical, mechanistic physics (plus some randomness).

It seems like thought and understanding have to be specific information flows in the brain

I don’t really see what else thought and understanding could be if not very very complex flows and processing of information in the brain. If you sit down to think about something, or understand something, what are you actually trying to produce? Usually you’re trying to change patterns in things outside your brain, like your mouth when you speak, your body when you move, or your hands when you write. The goal of thinking doesn’t seem to be to change anything about the physical stuff in your brain. The stuff inside your brain DOES change (synapses get stronger or weaker, and neurons change how easily they fire), but only to better shape the flow of information toward what you’re after. So it seems like whether our brains (or anything else) can think and understand comes down to the specific way information flows through them. This is not to say that anything could think if it could produce the same outputs from the same inputs as my brain. If I type “Hello, World!” and a computer runs the line of code print("Hello, World!"), we produce the same output, but obviously the code isn’t thinking. Thought and understanding depend on the specific ways information is processed inside a system, because that’s what determines whether it can do similar useful things with information in very new contexts, which seems like a clear baseline requirement for real thought. It’s also not to say that all information processing is thinking. A thermostat processes information too. The claim is that thinking is one particular, very rich kind of information processing, and the question for this post is whether things other than brains can do that kind.

If thought and understanding are about the flow of information, then they don’t depend on the specific substance carrying it (as we saw in the information theory section, information is substrate independent). The question is whether a given physical system can perform the causal roles needed for the kind of information flow required for thought and understanding. To figure out whether computers can perform these functions, we need to turn to computer science.

Some takeaways from computer science

In 1936, Alan Turing, often called the father of computer science, wrote a paper on what it means to “compute” something, and what could do it. At the time, “computer” was still a human job title, and he was trying to come up with a definition of a machine that could do all the same things a human computer possibly could. He proposed what came to be known as a “Turing machine,” which has a long strip of tape divided into squares that can each hold a symbol, and a head that reads one square at a time. The machine is always in one of a limited number of “states,” each of which is like one specific line of the instruction sheet. At each moment, based only on the combination of the state it’s in and the specific symbol in front of it, it writes a new symbol, moves one square left or right, and switches to a new state. The tape’s assumed to be infinitely long to allow the machine to get as detailed as possible. The reasons take a while to explain, but Turing suggested that any calculation a human could do by mechanically following step-by-step rules, with a pencil and unlimited paper, could also be done by this machine. This claim is now called the Church-Turing thesis.

Different Turing machines could perform different tasks. Solving a calculus problem is a step-by-step process, so is finding the optimal route to travel to a location. A system is “Turing complete” if it can simulate any specific Turing machine, which means it can carry out any calculation that can be written as step-by-step rules with enough memory and time.

Your laptop is Turing complete, because it can follow basically any set of instructions you give it, whether it’s running a spreadsheet or a video game with complex physics. The only two things that limit it are what you do with it and how much memory it has.

A lot of other things are Turing complete. For example, the trading card game Magic: The Gathering. In 2019, researchers found that you can set up Magic games where the creature tokens stand in for symbols on the tape of Turing machines and can be made to run any computational process. People have also built working computers out of the circuits you can make inside Minecraft.

A simulated computer could also be Turing complete. Emulators are simulated computers running on other computers. Many people use emulators on their laptops to play old Nintendo games that can’t be played on modern computers. The simulated parts of the computer move information around the same way a physical computer’s parts do, for the same reason people can build working circuits inside Minecraft.

Importantly, a Turing complete computer can simulate any process that follows the laws of classical physics to any level of detail, because classical physics can be written entirely as equations that a computer can work through step-by-step. Since the brain’s processes follow classical physics, a machine that’s Turing complete could in principle simulate a brain as precisely as we like. That gets us to premise 3 of my argument: a computer can simulate the brain’s information processing to whatever level of detail matters.

Even if the brain did depend on quantum physics, this could also be simulated, just much, much more slowly. It’s been shown that quantum computers can’t compute anything a Turing machine can’t, they can just do some kinds of computing much faster. The brain being classical mostly matters for how practical it would be to recreate the processes inside.

Would a simulated mind be able to really think and understand?

Many philosophers have debated whether a simulation of a brain would have phenomenal consciousness, whether there would be something it’s like to be the simulated brain. John Searle seemed pretty confident that a simulated brain would not have phenomenal consciousness, while David Chalmers believes it would.

I’m going to sidestep the question of phenomenal consciousness in simulated brains. As I argued in Part 1, I don’t believe that phenomenal consciousness is required for thought or understanding. What’s required is for a system to handle information in particular ways, using very high-level functions we don’t really understand in human brains yet. The question is whether a simulated brain could think and understand. A common line (going back to Searle) is that a simulated brain wouldn’t really think, for the same reason a simulated rainstorm doesn’t get anyone wet.

Importantly, I’m not claiming that AI models are simulated human brains. When you make an LLM, you’re not mapping the human brain and mimicking all the physical stuff happening inside. I’m instead trying to show that it’s possible in principle for a computer process to think, and will get to the details of how AI systems actually work in Part 3. If you’d agree that a simulation of a human brain with enough accurate detail could really think, you agree that computer programs can in principle think, and the question from there is just what computer systems qualify as really thinking. Right now I’m trying to get you to that first step, using an imagined perfectly simulated human brain to do it.

So far I’ve argued for premises 1 through 3. Thinking is something physical systems do, what matters is how a system processes information, and a computer can simulate the brain’s information processing to any level of detail. Together, I think these get us almost all the way to the conclusion. What matters for thinking and understanding is how information is altered by the system and affects other information in the system, and what the system can do with that information once it’s done processing. Whatever processes in the brain manage this information flow can all in principle be replicated by a Turing complete computer. If a simulation at one level of detail leaves out key subtleties in how those processes interact, it can just add more detail. And because the brain is so noisy (ion channels snap open at random, synapses fail), details far below the level of that noise can’t make any systematic difference to how it processes information (they just add to the noise that’s already there), so there has to be some finite level of detail that captures everything relevant to thought. A Turing complete computer can simulate the brain at that level of detail. The simulated brain would do the exact same type of information processing as the real brain, in the same way that a video game run on a good emulator (a simulated computer) plays exactly like it did on the original console. The simulation is simply another substrate that is handling the information. Because thinking and understanding are a specific type of information processing in the brain, and a simulated brain (like a simulated computer) could perform all the same types of information processing, computers can in principle really think and understand.

The last step is probably the most controversial. Is a computer simulation merely another substrate? That’s premise 4 from the start: a detailed enough simulation of an information process is that process, running in a new substrate.

To simulate a process on a computer is to set up the computer’s physical parts so that they carry the same information as the parts of the original process, and change in response to each other in the same pattern (at whatever level of detail you’re simulating). That’s also just a description of a process running in a new substrate. Earlier I defined a substrate as the physical stuff a process happens in. In a computer simulation of a brain, the process is the brain’s information processing, and the physical stuff it’s happening in is the computer’s silicon.

Let’s circle back to the imagined silicon neurons from earlier. Start with your brain after every neuron has been swapped for a silicon one. Now replace a small cluster of those silicon neurons with a single chip that takes the same inputs and produces the same outputs the cluster would have. Then replace bigger and bigger clusters, until one computer is calculating what every part of your brain would do. Every step changes how the information processing is physically carried out, but none of them changes the processing itself. At the end you have a brain simulation. If the gradual neuron replacement kept your thinking intact, a skeptic has to say at which of these steps it disappeared, and why.

In any other case that involves something that’s fundamentally information rather than matter, it’s clear that if you simulate something that handles the information in the exact same way, the exact same pattern of information emerges, and thus “the same thing is happening.” If you run a calculator program on an emulator, or in Minecraft, it really also calculates. Right now, you’re looking at words on a simulated piece of paper (your screen). Are these “real” words? Did I “really” write them? Yes, because words are fundamentally information, and specific symbols that communicate the information, that don’t depend on the specific substrate that hosts them. If the mind is also about information processing, all this same processing could happen in a simulated brain, and thus it would have the same qualities of mind we call real thought and understanding, for the same reason that you’re reading real words right now.

I think a lot of people’s immediate reaction to this is that the simulated brain wouldn’t have phenomenal consciousness in the way we have. Again, I’m leaving that debate aside, and would just ask you to recall the arguments I made in Part 1 that phenomenal consciousness doesn’t seem to be required for real thought and understanding. What matters is the right ways of handling information, which the simulated brain would have.

“Simulation” is itself just a name for another physical process. If we go back to the example of an emulator, a simulated computer, what’s actually happening is that we’ve basically just added an additional layer of programs and code to another program and code, that are all interacting with each other. There’s no deep difference between a simulation of a computer running a game, and running a game that just happens to have a lot of specific extra layers of code to it that it comes with. This is all ultimately just different specific causal relationships between computer code, which all come down to transistors in computer chips reacting to each other in specific ways, and it’s convenient for us to call a part of that code “a simulated computer” because it happens to be able to also perform the functions of a computer with other programs. These are just two separate methods of doing the exact same information processing, in the same way that the same type of boat can be made from different types of wood, or with a second layer of wood. If what matters for thought and understanding is how the brain works with information, not the specific atoms it’s made of, and a simulated brain would do the exact same things with the information, then what it’s doing seems to also be fully and completely “thought” in the exact same way the video game run on an emulator is fully and completely the same game.

Maybe the most famous objection to this is John Searle’s line about how simulated rainstorms don’t get anything wet, and simulated fire doesn’t burn down real neighborhoods, so why would a computer simulation of understanding actually understand anything?

This looks to me a lot like saying that the words you’re reading are just simulated words, so they can’t really convey information in the way a physical book can. If I’m right that thought is a kind of information processing, it’s mixing up what understanding and thought are. The nature of water and fire is fundamentally bound up with specific physical properties resulting from what they’re made of. But information is independent of any one substrate. These words can exist in my head and as electrical signals in my computer and radio waves to my router etc. Many simulated games have both things that we’d consider “real” and “fake” that are obviously differentiated by whether the thing depends on specific physical substance. Fire in Minecraft can’t actually burn anything, it’s fake, but if I build a calculator in Minecraft it can really do arithmetic, so it’s a real calculator. The realness of information processing clearly does not depend on what substrate is hosting it.

So everything comes down to which kind of thing thought is. Searle compared understanding to biological processes like lactation and photosynthesis. Photosynthesis produces sugar, a physical substance, but as I argued earlier, what thinking produces is patterns in your speech, your movements, and your writing, which is a lot more like what the Minecraft calculator produces.

Maybe real understanding requires a specific embodied interaction with the outside world, so it does rely on our physical bodies in a way computers can’t access. I’m very skeptical of this, because our interactions with the world are entirely mediated through systems in our body that convert them to the type of information our brains can handle: electrical and chemical signals in our nervous system. When I have an embodied experience of walking through a forest, the physical wood the trees are made of isn’t directly involved in my mental processing. Instead, what’s happening is that my body has receptors all over it (touch receptors in my skin, cells in my eyes that detect light, cells in my inner ear that respond to sound, etc.). This all seems obviously there to pick up on informational patterns light or sound or the texture of objects is carrying, and then convert it to the type of information patterns that the brain can work with, in the same way that while I can experience a lot of different patterns in the world that I can communicate in my writing, my wifi router can only work with pulses in radio waves, so the information needs to be translated into those before it can deliver them to you. The question then becomes whether a simulated brain could receive the kind of constant informational input that human bodies receive, and that seems like a difference of degree in access points to information rather than a difference in the kind of thing the simulated brain is doing.

Programs aren’t just in the eye of the beholder

A more sophisticated objection to the idea that a computer simulation of a brain could really think is that whether a computer is running any specific program at all is in the eye of the beholder.

In Borges’s short story The Library of Babel, the universe is a library containing every possible book of 410 pages composed from a fixed set of symbols. Its inhabitants search through overwhelming nonsense for books that explain their lives and their world. Any answer that can be written within the library’s format must be there somewhere, alongside countless false answers. But the narrator also suggests that apparent gibberish may carry meaning in an unfamiliar language. Taking this idea further, any book could be made to say anything, provided we were free to invent the language or code in which it was read. Any book could explain everything about your life if read with the correct language.

Similarly, any object, including rocks or walls, can be said to contain any computer program, if someone just used the right tools to read them in the right way. If I had the exact right scanner for a specific rock, where the scanner read the details of the rock’s surface as inputs, and I taught it to translate those signals in specific ways, I could determine that the rock contains full instructions for running the computer game Roller Coaster Tycoon, and play it off the rock like I would a CD. Searle argued that minds seem much more fundamental to the world than this, so computer simulations of minds can’t be real minds, because whether a simulation is doing anything in particular is just a story we tell about it.

The standard response to this, which David Chalmers worked out in detail in a 1996 paper called “Does a Rock Implement Every Finite-State Automaton?”, is that running a program can’t just rely on physical states happening to line up with a program’s steps. It requires the right cause-and-effect relationships, including what would happen if things went differently. A rock only “runs” Roller Coaster Tycoon in the specific sense that someone could run a very specific linear play through of the game from beginning to end. The rock lacks the responsiveness to enable the player to make any decisions or include any cause and effect relationships.

A simulation of a brain could easily respond to new causes with new coherent effects that a rock or wall couldn’t. If you showed its simulated eyes a different picture, its neurons would fire differently in the way yours would. Any argument that it’s still “fake” fails my brain test from earlier, where whether you yourself have a mind is an equally arbitrary matter of perspective.

There are a lot more philosophical arguments about the meaning of understanding and thinking and how it relates to simulated brains, most famously Searle’s Chinese Room. Before getting to those, I need to address a more basic common scientific complaint, that “the brain is not a computer,” and then build an argument against those other philosophical objections based on the fact that, the way they’re usually used, most of them would also rule out any physical system thinking, including the brain.

What do people mean when they say “the brain is not a computer”?

One of the most common ways people push back against what I’m arguing here is to say “but science says the brain isn’t a computer.” There are many popular articles making this claim:

• Robert Epstein’s 2016 Aeon essay “The empty brain,” with a subtitle that says “Your brain does not process information, retrieve knowledge or store memories.” That seems bad for my argument!

• Matthew Cobb’s Guardian article “Why your brain is not a computer,” which was adapted from his book “The Idea of the Brain”

• The neuroscientist Anil Seth has argued that the brain is not a computer, and that the idea that computation is enough for consciousness comes from taking that metaphor too literally.

These articles often get shared to say that AI cannot in principle think or understand. I’ve seen Epstein’s article shared a lot on Twitter recently. However, I think that even if every claim in these articles were true, it wouldn’t change my argument at all, except for two claims. One is that brains don’t process information, which seems obviously false. The other is that the brain is analog in a way computers can’t capture, which is the strongest version of this objection, and I’ll answer it below.

Look back at the five steps of my argument at the top. None of them says the brain is a computer. I’m saying something more like “The game Monopoly can be fully run and played on a computer. All relevant aspects of Monopoly can happen in a computer” rather than “The game Monopoly IS a computer.” Even if the brain is nothing like a computer, computers successfully simulate things that are not at all like computers all the time, like weather, and as I argued above, thought is the type of thing that is really truly thought if it is successfully simulated. Whether a computer can do what the brain does is a completely separate question from whether the brain is a computer, for the same reason that whether my computer can play Roller Coaster Tycoon is separate from whether Roller Coaster Tycoon is itself a computer that can run programs. Large language models aren’t computers either, in the same way Roller Coaster Tycoon isn’t. They’re things computers run.

Part of why this distinction gets lost is that the word “computer” actually has two different meanings that regularly get confused. In everyday speech, it means something like your laptop, that has a very specific architecture involving memory and hand-written code. In computer science, it has a more technical meaning that goes back to Turing, where it means any system that processes information by following a well-defined set of rules or instructions, regardless of its physical makeup. Our brains might be computers in the second sense, and obviously work very differently from the first sense. This is a good paper from 2022 arguing that the whole debate is so semantically confused that it doesn’t seem worth having. This is one reason why I’m interested in constructing an argument that doesn’t hinge on it at all.

Take Epstein’s essay, which has been going around a lot recently. His main point is that the brain doesn’t work like a laptop. It doesn’t store clear, distinct memories that can be easily retrieved, and it doesn’t follow clear internal written rules for how to use words. On most of this, I agree! But his picture of the brain is a much better description of current AI models than of the computers he’s contrasting the brain with.

Take this passage:

As we navigate through the world, we are changed by a variety of experiences. Of special note are experiences of three types: (1) we observe what is happening around us (other people behaving, sounds of music, instructions directed at us, words on pages, images on screens); (2) we are exposed to the pairing of unimportant stimuli (such as sirens) with important stimuli (such as the appearance of police cars); (3) we are punished or rewarded for behaving in certain ways.

We become more effective in our lives if we change in ways that are consistent with these experiences – if we can now recite a poem or sing a song, if we are able to follow the instructions we are given, if we respond to the unimportant stimuli more like we do to the important stimuli, if we refrain from behaving in ways that were punished, if we behave more frequently in ways that were rewarded.

As I’ll discuss in Part 3, AI models don’t store what they learn as separate files in memory that they can easily pull up the way your laptop stores information. The information they “learn” is spread across their weights connecting their “neurons” and it can’t be read off as clearly written rules. Like people, they can still memorize poems word for word with enough training, but this is very different from a computer storing the poem’s text in easily accessible memory. This passage describes current AI models so well that it’s strange to see it shared as evidence that AI can’t think.

The one claim in the essay that would actually hurt my argument is that brains don’t process information at all. Epstein seems to be using “information processing” in a much narrower sense than the formal one. He’s attacking what psychologists call the “information processing” model of the mind, where the phrase means storing, retrieving, and manipulating symbols, a lot like a laptop does. In Shannon’s sense, and information theory in general, information is anything that reduces uncertainty, and Epstein’s own description of memory, where learning a song changes your brain so you can sing it later, is a clear example of processing information. Learning the song reduces uncertainty about what notes to sing when. He also describes a student who can’t draw a dollar bill accurately from memory, as evidence that nothing like a picture of the bill is stored in our brains:

I agree with Epstein that this obviously shows that the mind doesn’t process information in the specific way a laptop does, but it also shows by the standards of information theory more broadly that the student clearly had a lot of information about dollar bills, just not perfect information. They could answer lots of yes or no questions that reduce uncertainty, like “Are dollar bills triangular?” or “Do dollar bills have ‘In God We Trust’ written on them?” His fly ball example is a good one too (outfielders really do follow a simple visual rule instead of calculating where the ball will land), but following a rule like that is still processing information about the ball. None of this affects my argument, which never assumed the brain stores perfect copies of anything.

A lot of people seem to be getting tripped up on this semantic confusion between the strict information processing model of the mind, where the phrase means storing, retrieving, and manipulating symbols like a computer, and the more broad definition of information from information theory. It’s undeniable that the brain processes information in the second way, and I agree with the critics that it doesn’t “process information” in the first much more narrow way. But the confusion between these claims is causing some people to make the bizarre assertion that what happens in brains is somehow not really information and not available to Turing complete systems, neither of which actually follows from the correct use of the term. People are mistakenly assuming that computers can only process information in the more limited, computer-like way of storing and retrieving code. In fact, Turing complete machines can do any form of broader information processing at all with the right programs running on them, and the type of information processing that happens in the human brain is a subcategory of that broader definition. This specific confusion has been a unique disaster for the AI debate.

Anil Seth’s commentary is more interesting and relevant to modern AI, but he’s talking about consciousness, and seems to separate it from questions of machine intelligence like I do. He has asked whether it’s possible to understand something without being conscious of it, and Part 1 was my argument that it is, so if Part 1 is right, his argument is also irrelevant to my claim. Maybe he’d agree? I’m not sure.

One last note on this is that a lot of people will often complain that the mind is always compared to the latest technology at the time. When we were in the Newtonian era, the mind was compared to clockwork, then when electricity and information technology took off it started to get compared to wires, then computers, then AI. I personally think each of these was actually a pretty useful metaphor compared to the common wisdom at the time, maybe with the exception of computers. It was a radical proposition when Hobbes suggested that humans are basically very complex machines in Leviathan. The mechanistic explanation of minds to this day draws a lot of ire and scorn despite being most compatible with our best science. Connections of wires transmitting information again looks like a much better explanation of minds to me than a lot of what was on offer in the 19th century. AI has renewed debates about how much of our own thinking is about prediction and pattern matching and carrying those forward, and again this is leaps and bounds better than the folk theory people carry around and get mad when you poke at it. I’m not really troubled by these metaphors, and insofar as they’re bad (like the computer metaphor reinforcing the idea of clear, easy-to-access concepts and memories stored in the brain), they are bad insofar as they reinforce our folk pre-scientific concepts of minds that they are otherwise helping us to move away from.

But isn’t the brain analog?

One other point that gets brought up a lot is that the brain is analog, in that it operates with a lot of continuous processes, whereas a computer being digital has to work in discrete steps. Anil Seth makes a point like this about consciousness. Maybe if thinking depends on infinitely fine-grained details of continuous physical processes, a digital simulation of it will never be able to model it exactly. It’s like trying to redraw a circle with lines that can only go up down left or right. You could approach the smooth curve of the circle infinitely but never fully get there.

But all real analog systems can only ever make use of a limited amount of precision, there’s always some random noise too. Your neurons can’t respond differently to two voltages that differ from each other by less than the random jiggling of the molecules around them, since that gets lost to noise. Anything below that is invisible to the brain’s information processing, so a computer program only needs to get down to this level of detail to perfectly mimic the information processing in the brain. There must be some finite level of digital detail that can capture everything relevant to thought in the analog systems of our brains.

A lot of arguments that computer processes can’t think are arguments that any physical system can’t think

Some of the most common arguments that AI can’t think seem to unknowingly reduce to “it’s just a physical system,” which would also apply to brains. If an argument applies just as well to your brain, it implies nothing can think or understand. This is the brain test I mentioned earlier.

The classic version of this argument came from the philosopher Leibniz in 1714. He imagined a thinking machine blown up to the size of a mill so you could walk around inside it. All you’d find, he said, are parts pushing on each other, and he took this as proof that perception couldn’t be explained mechanically. (The header art in this series is a jokey reference to Leibniz’s mill.) A lot of modern arguments against AI thinking look like versions of Leibniz’s mill. You can look inside an AI model, find only arithmetic and cause-and-effect relationships, and conclude that it’s not actually thinking. But a brain the size of a mill would also look to us like it wasn’t thinking, since it’d just be ions and proteins bumping into each other. The fact that we can’t see where the thinking happens when we look at the individual parts of a mechanistic system tells us nothing about whether the system thinks. Leibniz’s mill fails the brain test.

Two other common arguments that fail the brain test (assuming what happens in the brain is physical) are the claims that AI models are too mechanistic to think, and that they’re just a version of Searle’s Chinese Room. There’s also my argument in Part 1 that we often require AI to have a kind of introspection that seems physically impossible for humans to have.

When you give an AI model a prompt, every calculation it does is fixed by the prompt and the model’s weights. AI models have a randomness setting called temperature, and if you turn it all the way down to zero, the model always picks the single most likely next word, so the same prompt will give you the same answer almost every time. Because the chips don’t always add numbers up in the same order, tiny rounding differences can sometimes change a word, which acts like a little extra randomness. But that doesn’t seem to add anything to the model’s ability to think.

If the temperature is above zero, the only reason you get different answers is that a random number generator is choosing among different words, weighted by how likely the model thinks each one is. The AI model is really just a domino chain plus dice: a completely straight line from cause to effect, with some randomness thrown in if you’d like.

It seems obvious that dominoes and dice can’t think. A straight single line of dominoes from premises to conclusions can’t think and understand. A straight causal deterministic line from one given input to one specific output isn’t thinking along the way, and adding randomness to this process doesn’t make it more like thinking at all. In fact, it makes it even less like thinking because pure randomness doesn’t tell the system anything about the question it’s answering. That means that the AI model must not really think.

The domino-and-dice picture is in fact a pretty accurate description of how AI models work. Any AI model can be understood as an incredibly complex web of domino chains, plus some randomness, and nothing else. The problem is that this is also a complete description of every physical system, including your brain. As I argued earlier, every physical system is made up of unimaginably vast numbers of processes that are either mechanistic or random, and there’s no third option. Your brain is a physical system, and your thinking can be described by classical physics (plus some randomness). Suppose we asked you a question and recorded your answer, then rewound the universe to exactly the same state, atom for atom, and asked you again. You’d follow the exact same chain of cause and effect and give the same “output,” for the same reason a billiard ball hit at the same speed and direction always ends up in the same spot. The only way your answer could come out differently is if some random events (like an ion channel happening to open) went differently the second time. Your brain is dominoes plus dice too, so this argument fails the brain test.

The argument that AI models are too mechanistic to think therefore implies that no purely physical system can think or understand. I’ve given a lot of reasons in this post and the last one to believe that thinking and understanding rely on purely physical processes in the brain. Either our sense that thinking needs something beyond mechanistic cause and effect plus randomness comes from a vague folk theory about what happens when we think, or AI models are missing some third way of processing information that’s neither mechanistic nor random. The first seems much, much more likely.

Many arguments that AI models can never really think are based on the idea that their mechanistic cause-and-effect processing is basically a form of unknowing information retrieval. People usually back this up with two famous thought experiments:

• Ned Block’s “Blockhead”

• John Searle’s Chinese Room

Blockhead is a computer that simply has a canned response stored for every possible conversation. You say “How are you?” and it just matches this to a pre-written response it’s supposed to give: “I’m well!” This obviously wouldn’t really understand anything. Block meant this thought experiment to show that whether something is intelligent depends on how it produces its answers. So passing a conversation test like Turing’s doesn’t, on its own, show that a machine is intelligent. I agree with that.

The Chinese Room is a bit different. In a 1980 paper, Searle imagined a man who doesn’t speak Chinese locked in a room with a rulebook (a program) for shuffling Chinese symbols. People input questions in Chinese into the room. The man follows the rules, and he passes back answers that are indistinguishable from a native speaker’s, without actually understanding a word. Searle argued therefore that following any step-by-step program, however complicated, can’t by itself be what we mean by understanding what the symbols mean, because programs only work with the shapes of symbols and can’t access their meaning directly. Some people in the AI debate now use the Chinese Room as if the rulebook were just a list of canned answers, which turns it into Blockhead.

I agree that Blockhead doesn’t really understand, and neither would a Chinese Room whose rulebook was just a list of canned answers. But that’s only because they lack the internal causal processes that would let them respond to fundamentally new situations using the meaning of the words. If the rulebook could mimic literally all the causal processes required for thinking and understanding, the man and the rulebook system together would “understand the words” the way we do, even though neither does on its own. It’s the same reason my brain matter and the laws of physics together produce a system called Andy that understands words, even though my individual neurons and the laws of physics don’t understand. Here the room and book are my brain matter, and the man is the laws of physics. I’ll address the comparison to Blockhead first, and then the point the full Chinese Room is making.

My problem is with how the two thought experiments get used in the current AI debate. People correctly point out that AI models can be described as “mere information retrieval,” because some specific answer is waiting at the end of any possible prompt. With the temperature at zero, a given prompt can only go one way, and any deviation from that is due to randomness. So maybe models are like the canned-answer version of the Chinese Room, with the guy inside sometimes spinning a wheel to pick at random between a few similar responses he doesn’t understand at all. But any purely mechanistic system can be described this way. If you let a block slide from rest down a frictionless ramp of height h, it will always reach the bottom at one very specific speed (√(19.6h) meters per second, with h in meters). In this sense, its final speed was “canned” and merely “retrieved” by the input. If you ask someone a question, their brain follows classical physics to arrive at one very specific answer. The only things that can change it are randomness and new input from you or the world while they’re answering. The brain can only respond to any input with systems that are mechanistic or random, and our sense that it’s some secret third thing comes from just how much information it’s handling. It’s hard to even imagine what that secret third thing could be.

Every physical system’s outputs are fixed by its inputs (plus randomness), but that’s different from the outputs being written down somewhere in advance and looked up. Blockhead looks its answers up. Brains and AI models have to work theirs out. There are more possible 30-word prompts than atoms in the observable universe (even if you only used the 10,000 most common English words), so no model could have an answer stored for each one. Block himself noted that a real Blockhead could never be built for this reason. So if “retrieval” means the answer is fixed by the input, brains are retrieval machines too. If it means the answer was stored ahead of time, AI models aren’t retrieval machines at all.

I separately worry that part of the reason the Blockhead and simplified Chinese Room examples are so compelling to people is that most people believe that AI models literally just have a large collection of canned responses. In a Searchlight Institute poll from August of last year, 45% of Americans said ChatGPT looks up an exact answer in a database, and another 21% said it follows a script of prewritten responses:

Only 28% got this right, which might be partly because the options seem kind of poorly written to me. Still…

If your takeaway from the Chinese Room is that anything whose outputs are fixed by its inputs (plus randomness) is by definition not really thinking or understanding, then you don’t believe any physical system can think or understand. That means the mind must hinge on extra nonphysical stuff. My claim in these last two posts has been that thinking and understanding very likely depend on purely physical processes. This makes it look more likely that the intuition that “mere mechanism + randomness” isn’t enough for thought comes from a folk theory that dissolves when you poke at it. And in fact when people try to pin down what these purely rote cause-and-effect processes are lacking, they get very hand-wavy and defer to a strong felt sense that understanding requires something over and above roteness + randomness. As I argued above, that strong felt sense might come from evolutionary pressure that rewarded early humans for only treating carbon-based living things as having minds. That pressure itself comes from carbon forming stable bonds that hold up in water, so carbon-based molecules could work as replicators billions of years before brains existed. This felt sense seems like a bad guide to machine thinking.

Turning now to the actual Chinese Room thought experiment that isn’t just a restatement of Blockhead. Searle meant it as evidence that manipulating symbols can’t on its own produce understanding, so running a computer program can’t either. But “blindly following step-by-step rules that can be written in a book” also describes any physical system that processes information, including the human brain. Searle actually agreed that brains can be described this way. He said we’re machines ourselves, running any number of programs, but that the programs aren’t what make us understand. If the rulebook could somehow contain the rules for everything every physical process in the brain would do given a specific input, I’d argue the system running it could think the way a brain does. Searle considered this case in the same paper. He was responding to the idea that a computer simulating every neuron firing in a Chinese speaker’s brain would understand Chinese, and he imagined the simulation built out of water pipes instead.

In his other thought experiment, a man who doesn’t speak Chinese operates an elaborate system of pipes and valves, with each pipe connection playing the role of a synapse in the Chinese speaker’s brain. He receives inputs and follows instructions telling him which valves to turn on and off. Eventually, the entire system produces answers in Chinese that come out the other end. Searle claims that the man operating the system obviously doesn’t understand Chinese. The pipes, “obviously,” don’t understand, and so Searle thought it was absurd to say the man and the pipes together did. He then concluded that a brain simulation only copies the formal structure of the brain’s neurons and leaves out the brain’s ability to understand. Thus, biological brains can understand, but simulations of brains cannot.

I think the man is a distraction here. In the pipe system, he’s playing the role that the laws of physics play in a brain, moving things along according to fixed rules. Asking whether he understands Chinese is like asking whether the laws of chemistry understand English. It helps to remember that the Chinese Room is Turing complete. The man can follow any instruction, so anything a computer can do, the room can do too. Someone could write an extremely long book telling him exactly what to do to run Roller Coaster Tycoon (humans already write those instructions at each level a computer works at). He wouldn’t need to understand anything about Roller Coaster Tycoon to run it, and it probably wouldn’t even be clear to him what was happening. Nobody would say the game isn’t really running. Searle’s reply to this kind of point was that the man could memorize the whole system and run it in his head, so there’d be nothing in the room but him. But as I’ll get to in a second, that would mean memorizing something like 100 trillion connections and running them at the speed of a brain, which no person could do, and at that point he’d be simulating a second brain in his head, and as I’ve argued before, that simulated brain’s thinking would be identical to a real brain’s thinking, so it could also really think and understand.

In this thought experiment, Searle seems to be leaning on the idea that to really understand a word is to be conscious of it, to have a subjective phenomenal experience in your inner mental theater to use it correctly. He defended this directly elsewhere, with what he called the “Connection Principle,” which says a mental state can only really be about something if it could at least in principle become conscious. I don’t see another reason to say that the pipe system doesn’t understand the words it’s using. I’ve argued in Part 1 that this view of language and understanding as requiring this inner phenomenal theater seems wrong. It doesn’t cohere with physics, or with how words actually behave, or even with our own subjective experience when we use language. It also doesn’t appear “obvious” to me at all that this system wouldn’t understand the words it’s using. To argue for my own view that the pipe system doesn’t need extra-physical phenomenal consciousness to understand the words it’s using, I’ll paint a picture of what it’d actually look like.

There are roughly 100 trillion synapses in the human brain, and since the average neuron fires about once a second or less, a simulation like this would need up to around 100 trillion valve operations per second. This guy managing the pipes would either have to keep up that pace, following unimaginably detailed instructions about which valves to turn depending on what the water is doing, or (operating at a more human speed) each response would take millions of years. Even if each valve and its pipes fit in a box about 4 inches on a side, the whole structure would be a cube almost 3 miles on each side. Suppose he could keep up with the brain’s real speed. You could theoretically remove my brain, hook up my body to signals generated by the outputs of this water system, and I would walk around and interact with the world, and because Searle is assuming that this water system can actually perfectly mimic all the same information processing happening in the human brain, any informational input to my body, whether it be a sight or sound or touch or impulse from my body, would be taken in by this massive system, and it would give out the same information back that my brain would have. I would say and do the exact same things real Andy would, including sitting down to write this long blog post. It’s easy to lose sight of the fact that “having the exact same ability to merely process information” as the human brain would lead to systems that can also perfectly run human bodies to the point that they could type blogs and talk and interact with the world without anyone noticing. This version of me run by the water system would write the same blog posts about data centers and water and go on podcasts to talk about them, no one noticing he was using way more water than all data centers… This watery version of me controlled by the pipes would, I’d say, “understand” the world just as well as I do. Literally all the ways I can use language, it can as well. This seems like a point in favor of my belief that phenomenal consciousness isn’t necessary for understanding.

Similarly, if someone were in the Chinese Room with a sufficiently advanced rulebook, that could simulate all the processes in my brain with trillions of little individual decisions like this, they could perfectly accurately control my brainless body and I’d live out my life exactly as I’m doing now. I’d use words in all the same ways I do now. The fact that the man in the room is merely following instructions without understanding them tells us as much as the fact that the physical laws governing my brain don’t themselves understand what I’m doing, which isn’t much. Sufficiently advanced mere rule following can simulate any form of information processing down to an arbitrary level of detail, so I argue that the full Chinese Room rulebook can think and understand the words it’s using with some sufficient level of detail in the rules the book uses (which would add trillions of steps per second per input).

Searle himself anticipated this reaction in the same paper, and called the entity like the version of me controlled by the pipe system or room “an ingenious mechanical dummy.” This seems ridiculous to me. At this point, Searle’s “ingenious mechanical dummy” can do anything and everything humans can do with words. It could live a life identical to mine. It could write all these blog posts and go about its day. It seems at this point that Searle’s relying so heavily on the idea that “real understanding” depends on phenomenal consciousness that he’s willing to say a system that could do literally everything humans can do with words in any and all contexts doesn’t understand the words. At this point the disagreement itself seems pretty semantic to me. If AI is a Chinese Room, it can do literally everything human minds can do, and just happens to lack some additional thing we call “understanding” that doesn’t have any effect on how we behave.

Searle makes a lot more arguments about why computer programs can’t think, including that they merely follow step-by-step rules and lack the “intentionality” of what the words they use are about. I claim that his argument here cannot be made to work with any conception of the brain as a physical system, because all physical systems are reducible to mechanistic step-by-step cause and effect processes. His man in the Chinese Room is playing the role of the laws of physics. The laws of physics don’t need to be able to understand something for the system as a whole to understand. If no combination of these can think, the brain can’t either. Searle’s Chinese Room often looks to me like an argument for a nonphysical Cartesian theater where the true meanings of words can be grasped by something not merely mechanistic (and thus not merely physical). Searle would deny this. He would often say that humans are biological machines. But his demands for what minds can do seem to require something over and above the physical to make them work, and if you agree with my argument so far, it seems much more likely that this itself is a folk theory that masks the possibility that the processes that happen in our brain can be reduced to more simple physical processes.

If thinking is a kind of information processing, and nothing about carbon makes it special, then Searle’s pipe intuition is relying on exactly the two things this series has argued against. One is a sense that only biological stuff can think, and the other is the idea that understanding a word requires a conscious inner observer experiencing its meaning. This whole series so far has been an implicit argument against the view of language and understanding that Searle puts forward as “obvious,” and I hope that after making it this far you can see why I don’t think it’s obvious at all.

Semantic understanding needs to made of parts that don’t semantically understand

A lot of people seem to want to say both that physics is causally closed and that “mere information processing” in the broader information theory sense (rather than the much more specific “information processing” model of the mind, where brains work a lot like laptops) isn’t enough for true semantic understanding of words, where you really understand what words mean. I’m not sure what other physical processes could play this role. The only options for the building blocks of semantic understanding are rote cause-to-effect domino chains and randomness, and what those do in brains is either move matter around or process information. Again, I worry here that people are making one of two mistakes when they say that the mind is purely physical, but also that understanding needs more than information processing:

• They are mistakenly assuming that computers can only do the more narrow kind of information processing involved in following clear, hand-written rules stored in memory, whereas in fact a computer can in principle do all the information processing the brain does, by simulating the same systems the brain uses, which are very different from computers. This is again like saying that because Roller Coaster Tycoon isn’t itself a computer, it can’t be run on a computer.

• They are smuggling in extra-physical folk theories without actually exhaustively listing the very limited set of basic building blocks that physics provides for any complex system. Physics only offers simple mechanistic, “rule following” patterns, or randomness, and at the level of what a system does, information processing in the general information theory sense is the only option on the table besides literally moving matter around. Semantic understanding needs to be built from individual parts that each have no semantic understanding.

As I said in the section on what minds are made of, once we accept that anything that thinks and understands needs to be made of parts that obviously individually don’t, a lot of the thought experiments about the ridiculousness of pipes or Chinese Rooms or mills really thinking start to look more like someone looking too hard at the fact that the individual parts can’t think, and failing to look at the behavior of the system as a whole, which is the only place where understanding can possibly appear.

Conclusion

To return to my main argument:

• Premise 1: Things that think and understand are reducible to physical processes. There’s no extra nonphysical ingredient.

• Premise 2: What makes a physical system think is how it processes information, and what it’s made of only matters insofar as it can do that processing.

• Premise 3: A computer can simulate the brain’s information processing to whatever level of detail matters.

• Premise 4: A detailed enough simulation of an information process is that process, running in a new substrate.

• Conclusion: a computer process can, in principle, think and understand.

If you accept all 4 premises, you believe that thought and understanding are specific types of information processing that happens in physical systems. Information is independent of specific substrates as long as that substrate can carry the information and perform the cause and effect behavior required to process it. It would be very weird if there were something fundamental to carbon over and above the causal role it plays in larger systems that made it necessary to thought, and the only reason all minds seem to have run on carbon-based biological matter so far is that it was the only stuff evolution could act on to create complexity in the first place, which minds require. The next question is whether silicon can run the same causal processes required for the type of information processing that makes up thought. Silicon computer chips are Turing complete, which implies they can simulate any classical physical process to an arbitrary level of detail. The type of information processing in a human brain would require a lot of detail, but it relies on classical physics and doesn’t rely on infinitely small details, so it’s a finite amount, so all information processing that happens in a physical brain could in principle also happen in a sufficiently detailed and accurate simulation of a brain, and since thought is just the right type of information processing, this simulated brain could also truly think. Thus, computer processes can in principle think and understand.

So if you’re with me, you agree that computer programs can in principle think and understand. The next question that Part 3 will try to answer is whether AI models specifically are specific instances of computer programs thinking and understanding. This is a long stretch from simulated human brains. AI models are very different from human brains. But remember that octopus and crow brains are also radically different from human brains, and yet they also seem to be able to do what we mean by thinking. It’s on me now to argue that AI models have the same internal qualities that together qualify for this, which will require a deep dive into how current AI models work and how they might successfully mimic all the processes that make up thought and understanding, to the point that they are real in the same way calculators built in Minecraft are real calculators.

Introduction

Okay, first of all, I want to be clear that when I say AI models can “think” and “understand” I’m not claiming that they have minds like ours. People use “consciousness” to mean two related but different things, which the philosopher Ned Block calls “phenomenal consciousness” and “access consciousness.” Phenomenal consciousness is the first-person experience of what it’s like to be you. We usually think of this kind of consciousness as something more than information processing. There’s some unique experience that can’t be translated into a pure third-person account of your brain states. Access consciousness is information in your brain being available to lots of parts of your mind at once. This one IS fully describable in the third-person language of information flowing through a physical system. An AI could have access consciousness (sharing information across lots of its internal processes at once) without having phenomenal consciousness. There wouldn’t be “anything it’s like to be” it. Being that AI would “feel” just like being dead. I won’t assume or argue that AI has phenomenal consciousness.

I’ll make three claims, one in each post:

1. Our thinking and understanding mostly don’t depend on consciousness, and where they do, the aspects of (access) consciousness they depend on could in principle be replicated in AI models (This post)

2. Silicon computers could in principle think and understand (Part 2)

3. Current AI models seem to have the qualities that are necessary and sufficient for real thinking and understanding (Part 3)

These claims are controversial, but I’m pretty convinced by the arguments for each. My argument won’t rest on any one controversial belief about the nature of thinking or AI. Instead, I want to show that most leading theories about the mind seem to strongly imply both claims, or at least give them a high enough chance of being true that we need to take them seriously.

Importantly, I won’t argue that AI can do all the thinking we can do, just that what AI does at this point fits my definition of real thinking and understanding. I can’t think in the way Einstein could, but that doesn’t mean I can’t think.

You might think this is all just arguing over semantics, and it doesn’t matter what words we use for AI, because it can do what it can do either way. I think this is wrong, because most people who say AI can’t “really think” go on to infer a lot from that. They conclude it won’t be able to “discover anything new,” or that it’ll be permanently limited in how much it can change the world. If it’s “just linear algebra” it will remain a fun toy. If it can’t think, it certainly can’t “plan,” so it might not be especially dangerous. The most fundamental inference seems to be that AI will never do well when it has to carry a line of reasoning through to a case it hasn’t seen before. These are the situations where only the most general, subtle patterns from past experience can help, and only when a careful conscious person deliberately thinks them through. This, above all, is what a lot of people mean by “thinking,” as opposed to regurgitating information. Getting this question right matters a lot for making sense of where AI is headed.

I’m also worried that as debates about AI have heated up, a lot of people have adopted a rule of thumb that any talk of AI “thinking” is effectively assuming that there’s a ghost in the machine, like assuming a radio has a little band inside it playing the music. This seems like a wild overreaction to me, and one that wouldn’t make sense outside our current cultural moment with AI. I’m pretty convinced that if I took a current model like Astra or Opus 5.5 back to 1960 and let people play around with it, the reaction from serious people would be “Wow! You’ve created machines that can actually think! They understand the words they’re using!” Alan Turing predicted in 1950 that by the end of the century, “the use of words and general educated opinion will have altered so much that one will be able to speak of machines thinking without expecting to be contradicted.” Today, people are much more wary of saying this, partly because they think “anthropomorphizing” AI buys into the hype or is a sign that you’ve developed psychosis. I think that’s an understandable (initial) reaction, but it’s starting to cause people to make serious mistakes. Part of my argument will be that the hard rule against “anthropomorphizing” AI is itself philosophically confused, and potentially anti-scientific, because it often implicitly depends on and enforces pre-scientific folk theories about what’s special about human minds.

It’s bad and dangerous to use anthropomorphic language for AI when it leads people to believe that AI has qualities it doesn’t. If someone comes to feel an AI really cares about them, or that it has a human-like mind, that seems pretty destabilizing, and it’s understandable that people worry about it. But the worry can go too far.

Before machines existed, humans and some animals were the only things that lifted objects. Once we built machines to help, it wasn’t bad to use anthropomorphic language and say a forklift “lifted” something, even though lifting used to be something only conscious, biological animals did. The word “computer” used to be a human job title, but no one today would say it’s anthropomorphizing a laptop to say it “computes.” These are obvious extreme examples. Where’s the line between clear cases like these and cases where it becomes bad to use anthropomorphic language? The obvious answer to me is that the line is where this language implies AI has a lot of abilities it doesn’t have. But the same line also shows where it’s bad NOT to use anthropomorphic language for AI, which is when avoiding it implies that AI lacks abilities it actually has. Ultimately, what makes something human-like is a specific combination of lots of simpler properties that aren’t human-like on their own. Anthropomorphic language is useful because it helps us simplify, understand, and predict the behavior of our fellow incredibly complex systems we call humans. It doesn’t pick out some fundamental unifying aspect of the properties of our minds that walls them all off from anything else in the universe having them.

So deciding when to anthropomorphize AI depends on what abilities it actually has, but also on what abilities our own minds have, and I think most people’s picture of how their own minds work is deeply mistaken. I’ve called this picture “folk-Cartesianism” in past posts. This view is that a lot of aspects of the mind come in a single inseparable package, held together (and walled off from machines) by the fact that all of them require a conscious, first-person perspective observing and managing them. This view imagines the mind as an internal movie theater, with a little version of you inside watching the movie of your life. The philosopher Daniel Dennett called this idea the “Cartesian theater” after Descartes, who often wrote about the mind as an unchanging observer (the “I” in “I think therefore I am”) watching its own experiences and thoughts directly.

The folk theory of mind says that when you “think,” you’re consciously observing individual concepts and moving them around the screen in your mind. You react to them and decide the direction of your thought. A key part of thinking (in this view) is powerful, direct introspection, where you turn your mind’s eye back on your own thought process and observe it directly. Your thoughts and inner experience are clear and directly accessible to you. You can observe them as if they were all written out on a piece of paper you were reading.

The folk theory says that if something can’t do this (if it can’t look at its inner mental screen and see its own thought process laid out clearly), then its “thinking” is basically only a domino chain running from premise to conclusion, like a rote computer program chugging along with no idea what it’s doing. That can be useful, but isn’t thought.

I believe that this picture is a big reason so many people object to the idea that AI “thinks.” If you hold it, “AI can think” smuggles in two drastically wrong (according to the critics) ideas at once. The first is that AI has first-person conscious experience. The second (which follows from the first) is that whatever AI does that looks like thinking is just as good as real human thought, when without that inner glow of consciousness it would have to be missing most of them.

In this post, I’ll argue that this folk theory is deeply wrong, for a few overlapping reasons. In Part 2 I’ll address a lot of popular philosophy (and folk) theories about why computers cannot think and understand, and give my reasons for why I think they’re wrong. In Part 3, I’ll go through what current AI can do. I’ll argue that current AI does enough of what we mean by thinking and understanding that the burden should now be on anyone who says it doesn’t think to say what’s still missing. And after this post, I hope you’ll agree that the list of what’s missing doesn’t include a Cartesian theater (a conscious observer inside the AI with direct, easy access to its own thought processes and control over them), because I don’t think humans have one either. I’ll knock down human thought a bit and raise up AI thought a bit, to the point that they might come close to meeting in the middle.

To be clear, I don’t mean to say here that AI can think all the thoughts humans can, for the same reason that when I say I can think, I don’t mean that I can do everything that any human mind can do. But a core part of how I think about the world is that most of our beliefs about our own minds and thought are useful but wildly imperfect folk theories and heuristics that get at reality about as much as saying “the rock wants to move toward the ground.” I think it’s a gigantic mistake to assume we have total access to what’s going on in our minds, and to let folk beliefs about our minds’ specialness override any debate about them.

How the human mind is different from our conception of it

Some examples of simple thought and understanding that don’t require consciousness

I’ll start with some simple, first-person tests you can run on your own thinking, to show that it’s not entirely like an inner theater. If your conscious mind does all your thinking and has full access to it, it should be pretty easy to watch it happening.

First, think of a city, any city at all.

Do you have one? Why did you pick that one? Do you know?

When I do this, I don’t consciously search through a mental list of every city I know and pick one. Instead, a city just appears, or several appear and I pick one of them. This seems like a pretty clear sign that some part of your brain understood what “city” meant well enough to search your inner catalog of examples without your consciousness overseeing it.

Now we’ll do another, and this time you should pay really close attention to the process in your head.

Pick an ice cream flavor.

Candidates appear, but again it’s not clear where they’re coming from.

Now finish this sentence: “The early bird catches the...”

I don’t think you had any mental control over whether you thought “worm” there. It appeared before you made any conscious decision, and I think it would be very hard to decide not to think of “worm.”

Think about the last time a word was on the tip of your tongue. You might be trying to remember an actor’s name, and can recall their face, the movies they were in, even the letter their name starts with. Then the answer suddenly appears hours later while you’re doing the dishes. Some mental process was humming away without your conscious mind knowing, and was suddenly able to produce a result and deliver it to your conscious mind.

All of these are pretty obvious evidence that retrieving information (which itself depends on understanding a huge amount of context about what words mean and how they relate to each other) can happen without our conscious minds observing it or making any decisions about it.

Folk theories of the mind often imagine language as playing a pretty secondary role to thought. When you speak, the folk theory imagines that you have some clear, pre-linguistic picture in your head of exactly what you want to say, and choose the words that will best communicate that picture to another mind. The full idea is clear before you speak, and language is just a useful bridge between minds, rather than something that our mental pictures depend on.

I’ll challenge this folk idea of language in a lot of ways in these posts, but one immediate problem with it is that the words we say often require a shocking amount of specific reasoning and understanding that doesn’t exist in our conscious minds before we say them. To see this, you can choose a topic you know a lot about, set a timer for 3 minutes, and then start speaking about it as much as you can. As you talk, try to pay really close attention to what’s happening in your conscious mind, and what decisions you’re actually making before the words come out of your mouth. You might be surprised to find that you don’t really know how a sentence will end when you start it, and there’s very little conscious planning happening at all. And yet you speak coherently, and can even learn from the things you hear yourself say. E. M. Forster tells a story about an old lady who asked, “How can I tell what I think till I see what I say?” This seems to imply that a lot of the “thought” required to plan and produce a complex paragraph happens outside your conscious awareness. And because producing it takes a deep understanding of the words and how they fit together into coherent English, as well as the concepts and how to best explain them via carefully structured sentences, a lot of that “understanding” is happening outside your conscious experience too. The fact that this works is one reason why writing can often be useful for learning. Writing can reveal full thoughts we’ve had that we might not have been consciously aware of.

One famous example of complex thought happening outside consciousness comes from the mathematician Henri Poincaré. He’d spent about 2 weeks working hard on a new kind of function. He took a break to go on a geology trip and stopped thinking about the problem. He says:

At the moment when I put my foot on the step the idea came to me, without anything in my former thoughts seeming to have paved the way for it, that the transformations I had used to define the Fuchsian functions were identical with those of non-Euclidean geometry. I did not verify the idea; I should not have had time, as, upon taking my seat in the omnibus, I went on with a conversation already commenced, but I felt a perfect certainty.

When he got back, he checked it, and it was correct! He concluded that sudden flashes of insight like these are “a manifest sign of long, unconscious prior work.”

In all of these examples, your conscious mind only gets access to the output of thinking. It doesn’t observe the processes that produced it. Scientists disagree about how far this goes. The neuropsychologist Karl Lashley had an extreme take in the 1950s, when he wrote “No activity of mind is ever conscious.” The psychologist George Miller later said, more mildly, that it’s “the result of thinking, not the process of thinking” that shows up in consciousness on its own.

So the folk idea that all thought and understanding require consciousness seems wrong. The question is what role consciousness plays in higher level thought and understanding.

A lot of your understanding happens outside of consciousness

So far my examples have mostly been loose associations (like thinking of a city) or drawing on things you already know (like talking about a topic). Maybe a lot of understanding and thought is by definition more complex than this, and requires conscious experience.

But we also understand much more complex things that we’re almost never conscious of, and never consciously learned. For example, English speakers put adjectives in a very consistent order. “A big red balloon” sounds normal, but something sounds strange about “A red big balloon.” The pattern adjectives follow is roughly opinion, then size, age, shape, color, origin, material, and purpose. I could describe Washington DC as a pleasant small old square gray city, and that sounds fine, but saying it’s a gray square old small pleasant city sounds strange and unnatural. You understand and use this pattern every time you speak, but you might never have been conscious of it, even when you “learned” it from years of hearing English. I would say I understand how to use English well, and this pattern is a key part of using English well, yet I went a long time not being conscious of this specific rule.

You also understand which words ought to follow others, well enough that your brain reacts to a strange word in a sentence within a fraction of a second, without you consciously deciding to check whether it makes sense. Neuroscientists have recorded people’s brain activity as they read sentences one word at a time. Some sentences ended with words that didn’t fit, like “He spread the warm bread with socks.” Strange words like “socks” trigger a distinct electrical response in the brain that peaks about 400 milliseconds after the word appears. Later experiments found that words that make complete sense also produce this same response, and the electrical response gets bigger the less expected the word is. So as we read, our brains seem to be constantly tracking how predictable each next word is. One explanation is that our brains use context to anticipate likely words. This seems like a core part of how we understand what we read. It’s subconscious (we’re not consciously weighing every word that could come next), and it looks surprisingly similar to how modern AI “predicts the next word.” This also looks to me like a deep understanding of how lots of possible words relate to the situation at hand, all happening subconsciousnly.

Some researchers wanted to see if there were any signs of deeper similarities between how our brains predict words and how AI does. AI models also assign a likelihood to every possible next word (that’s how they write), so we can see when a word shows up that the model considers very unlikely, like spreading bread with “socks.” Several research groups have found that these likelihoods inside language models reliably predict how strongly people’s brains react to more or less expected words. And the models that are best at predicting the next word tend to be the best at predicting the brain activity of people reading. So AI seems to successfully mimic at least this part of our subconscious understanding.

Our introspection and memory of our conscious thought are fallible

So we’ve seen there are types of thought and understanding that don’t occur in our conscious minds, which implies they don’t require consciousness. But the folk picture also runs into trouble with the thinking that does happen in our conscious minds. Our very recent mental past is surprisingly opaque to us. Even our honest explanations of our own recent conscious decisions can often be false.

The most extreme example of our beliefs about our own minds being false comes from split-brain patients, who have had the main bundle of fibers connecting the two halves of their brain cut to treat severe epilepsy. Researchers can show these patients a picture that only one half of their brain sees. In most people, only the left half of the brain can generate speech. In a famous study, the neuroscientist Michael Gazzaniga showed a picture of a chicken claw to the left side of a patient’s brain, and a picture of a snowy scene to the right side. The patient was then asked to pick related pictures from a set. The patient’s right hand (connected to the left side of his brain, which had seen the claw) chose a chicken, and his left hand chose a snow shovel. When he was asked why (remember, this is the left side of his brain talking), he gave an answer he honestly believed. He explained that the chicken obviously went with the claw, and “you need a shovel to clean out a chicken shed.”

So the speaking part of him had no idea why his other hand picked the shovel, but it came up with a compelling story, after the fact, for why he’d consciously chosen it. Gazzaniga came to believe that we all have a system like this in our brain, which he calls the “interpreter.” Its job is to come up with after-the-fact justifications for what we do, which we experience as honest beliefs, and to weave them into a coherent story about us, whether or not it has access to the real reasons behind our decisions. If this is right, the reasons behind even our very recent conscious decisions are more hidden from us than they seem, and consciousness plays less of a role in our higher-level decisions than we’d expect.

This hints at a broader claim I want to make, that introspection is just another mental process. It takes in uncertain information, processes it, and gives an output that’s as reliable or unreliable as our beliefs about the outside world. Ultimately, our thoughts are the result of activity in our brain. There isn’t some ethereal realm we exist in where we can observe our thoughts directly. Our beliefs about our own inner lives are fallible stories about the past, made more coherent by the (often self-serving) narratives our brains offer up, which we honestly come to believe. The idea that introspection is some transcendent act of turning the mind’s eye back on itself and seeing its own processes completely clearly is pretty suspicious.

We’ve seen similar evidence for something like an “interpreter” in people with intact brains too. In a 2005 study, people were shown pairs of photos of women’s faces and asked which one they found more attractive. In some trials, the researchers used sleight of hand to swap the photos, so people were handed the face they’d rejected and asked why they’d chosen it. The swaps were only caught in the moment 13% of the time, and only a quarter of the swaps were noticed at any point in the process. Instead, people very confidently and honestly explained why they had “chosen” this face, often indicating features that only the swapped face had. For example, one man said he had chosen the “more attractive woman” because she was wearing earrings, but the woman he had actually chosen wasn’t wearing any. The woman he had been handed after and was looking at was wearing them. 84% of the people who didn’t notice a swap said they thought they would notice one.

So people can honestly come to believe fake narratives their brains offer up about very recent conscious decisions. Dennett believed that this process describes our whole minds. He didn’t think there’s actually a single place in the brain where everything comes together for a viewer to see. He believed the self is basically just the broad story all these individual narratives and justifications add up to, which he called a “center of narrative gravity.” We tell stories about ourselves, but “for the most part we don’t spin them; they spin us.”

This view of introspection makes sense if you think about what actually happens when we introspect. Whenever we think at all, we’re feeding information about our own past behavior back into our cognitive processes.

The output of mental activity can then be used by other mental activity.

All of our mental activity is fallible, and takes in uncertain information about what’s happened before. Introspection specifically takes in past (fallible) evidence we have about our own behavior and computes a (fallible) output.

What doesn’t seem possible is a folk theory of introspection that imagines us as floating observers with a god’s-eye view of everything happening in our own minds:

Ultimately what’s available to the mind is the results of past computations and mental processes. Some of those could involve reporting back to the mind a lot of the individual steps taken. Introspection in my view is like examining any other aspect of the world. This more natural kind of introspection seems easily available to AI, which could also just collect information about what its past processes did and act on it. It doesn’t need a floating eye in the middle of its “experience” looking back at its own processes.

Some people point out that AI models can only move information forward through their layers, and take this as a sign that they can’t do what we do when we introspect. To me, this sounds kind of like saying that human minds can’t introspect because all the processes inside are moving forward in time. I think our folk theory often gives us the mistaken idea that we can directly peer into the recent past of our own mind and examine what’s there. What’s actually happening is that we’re using current mental processes to retrieve information about past ones. AI models can do the same thing, retrieving the results of earlier processes while answering a question (and we know they do this regularly). Once you think about introspection this way, it’s a little hard to see why current AI models couldn’t do something similar.

So it’s looking bad for the ‘little version of you in a theater’ view of the mind. The process that produces your thoughts (which takes understanding) is hidden from your conscious mind, and the process that explains your thoughts to you might be a separate “interpreter” without reliable access to the first one.

What about slow, careful thinking?

Of course, most of my examples so far are quick, automatic kinds of thinking. Slow careful reasoning, like working through a philosophy argument or doing math, looks like a much stronger candidate for where the conscious version of you takes over.

Even here, lots of individual steps happen subconsciously. If you ask me what 7 times 8 is, 56 just appears in my head. Maybe slow thinking works by summoning these opaque individual answers into our attention, and then examining them in the light of consciousness to build a deeper understanding.

Slower reasoning is not our default. It takes more energy and often feels unpleasant (this was the big theme of my last post on EA). In the famous simple problem:

“A bat and a ball cost $1.10 together, and the bat costs a dollar more than the ball. How much does the ball cost?”

The math looks so simple that most people don’t even think to slow down, and only switch into careful reasoning after they hear the ball isn’t $0.10 and get confused.

What actually determines when you enter this slow thinking? Is it a conscious decision? You can definitely choose to think more slowly and deliberately about a problem, but often the switch happens subconsciously. We don’t start reasoning slowly until something “snaps us awake,” like coming off autopilot. A lot of the time, it’s not clear to my conscious mind what aspect of me causes that slower reasoning to kick off.

But once we’re in slow reasoning mode, this does seem like where consciousness matters most for our thinking. It’s hard to picture a machine doing our most important kind of slow, deliberate thinking (pulling in information from lots of contexts at once and carrying very general patterns into new areas) without something like consciousness. So I now need to say what consciousness is (or might be) and what role it seems to play in our most careful thinking.

What phenomenal consciousness is

I’ll rely on Ned Block’s distinction from the introduction, between phenomenal consciousness (the first-person, what-it’s-like experience Thomas Nagel writes about in “What Is It Like to Be a Bat?”) and access consciousness (information being broadly available to lots of processes in your brain).

I am personally pretty convinced that phenomenal consciousness is not required for thinking or understanding. Access consciousness is required, but I think the way it works in humans implies current AIs probably have something close enough to it to allow for high-level thought.

Most of this section will be on phenomenal consciousness, and the next will be on access consciousness.

I think most people see phenomenal consciousness as the clear ultimate barrier between human minds and machines. No matter how advanced machines get, they will never ascend above information processing and have the first-person subjective experience we have of the world. For example, a machine could look at this square, and detect the wavelength of the light it’s giving off:

But it could never have the first-person experience of the redness of the red.

When you look at this square, I could write a third-person description of what’s happening in your brain. I could record the activity of every neuron, the wavelength of light, the associations you’re drawing between the color and past information your brain’s received about apples or sunsets. But no amount of third-person information about your brain, even the position of every particle and every electrical signal, could ever give me the first-person experience of being you seeing the red, the redness of the red over and above the information carried by the light.

This is what the philosopher David Chalmers called the “hard problem” of consciousness, the strange fact that consciousness alone seems to be inaccessible to third-person observation. Nothing else in the universe works like this. Everything else exists somewhere in space and time, follows regular physical laws we can write down as math equations, and can be measured by anyone with the right instruments. More complex “things in the world” like the national debt can ultimately be reduced to the behavior or expected behavior of systems made up entirely of things that have these basic qualities. First-person conscious experience seems to break all these rules. People could dig around in your brain as much as they wanted and find the neurons that respond to red, but it doesn’t seem like they’d ever find the redness.

This seems pretty hard to deny. Unfortunately, another fact about the world also seems really hard to deny, and it contradicts the first one. Physics seems to be “causally closed,” meaning the only things that affect physical objects are other physical things described by the laws of physics.

One of the main reasons to believe this is the conservation of energy, which says that when energy leaves one object, it always goes to other objects. When one type of energy disappears from an object without going to another object, it’s just turning into another type of energy. For example, when a rock falls, its gravitational potential energy goes down, and its kinetic energy (the energy it has as a result of moving) goes up. If energy is never created or destroyed, then no energy can be added to the world from some mysterious realm beyond physics. If nonphysical things were affecting the physical world, objects would gain or lose energy or momentum out of nowhere, breaking these laws. If a ghost that exists outside physical reality picked up a rock, the rock would gain energy that came from nowhere. Classical physics couldn’t work if violations like this were happening all the time, so physicists came to assume they weren’t happening at all.

We now know that classical physics was way too simplistic, and its version of these laws IS regularly violated. Mass and energy aren’t conserved separately. Mass can turn into energy, and energy into mass. Physicists now believe in the conservation of mass-energy. You can turn about a gram of mass, roughly the weight of a dollar bill, into the energy released in the Trinity test (the first atomic bomb), but you can’t turn it into the amount of energy in the sun. There are still strict rules for these conversions, set by E = mc² and the broader conservation law.

On the standard textbook reading of quantum mechanics, there’s also fundamental randomness in the world that isn’t the result of our not having enough information. If you rewound the universe, started it from the same initial conditions, and played it back, different things could happen. This is different from the everyday probabilities we deal with. When I roll dice, they’re so completely governed by classical physics that if you knew their exact position and velocity, their distance from the table, and exactly how they’d bounce off it, you could predict with perfect certainty what numbers they’d land on. If you rewound the roll and rolled them in exactly the same way, they’d land on the same numbers every time (unless quantum effects showed up at the everyday scale, which is extremely unlikely but not impossible). Our everyday uncertainty just comes from missing information. Quantum mechanics in comparison appears to be truly probabilistic.

Some people mistakenly think that quantum randomness opens the door for nonphysical entities to affect the world. The idea is that something like consciousness, existing beyond the laws of physics, could take an event with a 60% chance of happening and “choose” that it happens. From the outside, this wouldn’t look like a violation of physics. But this deeply misunderstands what quantum probabilities are. For the rest of quantum mechanics to work, a 60% quantum probability has to reflect the true odds of the event. The odds themselves are part of physical law. An outside entity meddling with them would basically be changing the 60% to 100%, which violates physical law in the same way as the ghost lifting the rock. Among other problems, the ability to change these probabilities would mean we could use entangled particles to send messages faster than the speed of light, breaking another core rule of modern physics. So quantum mechanics is actually just as walled off from supernatural effects as classical physics was.

Physics is (I’m biased) our very most successful field of science. It has broken down the nature of ultimate reality and successfully predicted the behavior of distant stars and black holes. Giving up causal closure would break a lot of what makes it work.

So, all that to say, physics looks really really causally closed to me. That leaves us with a problem, because conscious experience doesn’t seem to behave like anything physical. All this means that of these three obvious-seeming statements, one has to be false:

1. Physics is causally closed. Everything that happens in the physical world is fully caused by other physical things (or, where there’s quantum randomness, has its odds set by them).

2. Phenomenal conscious experience isn’t reducible to physics.

3. Phenomenal conscious experience affects what we do, including what we say about it.

Which one you cut matters a lot for how consciousness affects thought, and so for whether machines can think. I’d say my main problem with human existence itself is that cutting any of these looks like complete garbage to me.

The decision to cut 3 is called “epiphenomenalism.” This is the idea that consciousness is not reducible to physics, but physics is causally closed, so consciousness doesn’t actually have any effect on the physical world. Our conscious experience is like a passive observer in a theater helplessly watching our lives unfold.

This can sound intuitive at first, but I think it looks completely crazy the more you consider it.

This view implies that whenever you say “I’m not just processing information, I actually experience the redness of red,” your experience of the redness of red played no part in making you say it. Your physical brain, your speech, and your writing (and anyone else’s) have never been affected at all by the fact that you have this special experience. It’s just a pure coincidence that our physical bodies and brains, which are basically machines without this phenomenal experience on their own, happened to start writing about this thing that has never once affected them but definitely does exist beyond the physical world. You may have heard of “philosophical zombies” (or p-zombies), an idea that fits naturally with this view. A p-zombie is physically identical to a normal person but doesn’t have phenomenal consciousness. There isn’t anything it’s like to be them. An epiphenomenalist believes an entire world could exist exactly like ours, with everyone doing everything they normally do, except that everyone’s a p-zombie. That’s easy for an epiphenomenalist to imagine, because our own world is basically that plus some extra stuff beyond physics, caused by the physical world but never causing anything in it.

I find this so strange that I can’t entertain it. Smarter people than me do entertain it, so take my reaction with a grain of salt. But after 15 years of wrestling with the idea it remains the theory of phenomenal consciousness I give the lowest odds of being true.

As a well-behaved physicist, I’m also very hesitant to cut the first option. There are lots of other problems with cutting it. Evolution gives us a lot of reason to think we’d see ourselves and other people as monumentally important compared to the rest of the world, so it’s a little suspicious that the one place our most successful scientific theory breaks down is inside our own heads.

I can see myself believing this if I were religious. If you are religious, and your belief that the mind transcends physical reality is why you don’t believe machines can think, my only ask is that you say that directly in conversations about it. I often find that debates about minds and machines can meander in meaningless details for hours before one person flags “Oh by the way, the main reason I’ve been denying your position is my religious beliefs, which I’m only mentioning now.” That clarifies where the actual disagreement is.

If you’re not religious, there are a ton of additional problems with cutting Option 1. Evolution by natural selection is a purely physical process. How did merely physical animals at some point start accessing, affecting, and being affected by some ethereal realm or substance beyond physics? Why are they the only entities in the universe that can access this? If you cut the first option, you’re not religious, AND that’s why you don’t believe AI can think, I think the burden of proof is squarely on you to explain how your view fits with a broader scientific picture of the world.

If you do cut the first option, an interesting question is how much of our thinking and understanding happens outside the physical world. Do we have thoughts that happen outside of our brains? If so, why does severe brain damage also impair specific kinds of thinking? Whatever role the nonphysical part of consciousness plays, our slow, deliberate, higher-level thinking still depends massively on the physical brain working in specific ways. That leaves open the possibility that machines without the inner spark of phenomenal consciousness could think this way too.

The last option to cut is 2, where we say that the sense that consciousness is somehow above physics is some kind of complex illusion. This is the one I lean toward, but only because the other two look so much worse. There’s a lot that troubles me about this view, most of all:

Come ooooon! It really really really really seems like there’s something happening here that isn’t just information processing. There’s a “way it is like to be me” that can’t be articulated with sufficient physics equations. Aaaaah!

But phenomenal consciousness is incredibly strange. There are ways it doesn’t cohere at all with our folk theories of it. It’s strange enough that I can at least entertain the idea that it’s a complex misunderstanding happening in an information-processing machine called my brain. To build your intuitions for how this could be possible, I’d very strongly recommend Dennett’s paper Quining Qualia, one of the most fun and accessible papers in philosophy of mind. Another good essay is Joe Carlsmith’s Grokking Illusionism.

This view, that phenomenal consciousness is reducible to physical processes, has many flavors, like illusionism. However weird you find this idea, epiphenomenalism is just as weird, and it adds a bizarre extra entity that doesn’t behave anything like how I experience consciousness. For example, whatever we mean by consciousness, I personally mean the thing that causes me to see that red box and type the words “Aaaaah!” But epiphenomenalism tells us my typing that was not at all caused by the redness of the red.

There are also philosophers who, when faced with this trilemma, throw up their hands and announce that our concepts of the world are simply not built to understand consciousness, and the answer here might be stranger than we can conceive.

How does any of this matter for whether AI could at some point think? Well, which story we believe tells us something about how phenomenal consciousness interacts with our thinking.

If you believe in epiphenomenalism (that phenomenal consciousness exists but has no causal power), then our having phenomenal consciousness has never affected the world at all. We could lack it and the world would look exactly the same. And humans can think and understand things. So on this view, whether AI could think and understand doesn’t depend at all on whether it has phenomenal consciousness, because our having it doesn’t matter to our thinking either.

If you believe consciousness is nonphysical and does affect the physical world, I’ve said why I think the burden of proof is on you, and even then most of our deliberate thinking still seems to run on the physical brain.

If you believe (like me) that phenomenal consciousness is probably entirely reducible to physical interactions, then whatever role it plays in thought could in principle be replicated by other physical processes. This is necessary but not sufficient for showing that AI could have it. Consciousness could be a special kind of physical process that depends on the substrate (the material the mind runs on). This view is currently defended most prominently by the neuroscientist Anil Seth. But Seth (like many others who hold this view) believes intelligence and consciousness can come apart, so machines could “think” and “understand” without being conscious. The question for people who hold this view becomes whether machines can do the correct type of physical things that human brains do, not whether there’s some extra-physical spark inside the machine.

If I consciously notice a mistake, and that lets me correct an argument or line of thought, something has changed in how my brain is processing the argument. Relevant information has been successfully introduced. We can look into what’s changed, and how it’s improved my thinking, seemingly entirely via information processing, without ever explaining what it felt like to notice it. If phenomenal consciousness is not reducible to access consciousness, this makes it look like one specific way information is introduced to my mental system, and anything it introduces that can’t be fully reduced to information (the redness of the red etc.) doesn’t seem to play a direct role in thought.

The fact that “I am experiencing the redness of red,” can be reducible to information the brain can use (I can write it out as a sentence that you can read and make sense of) but the phenomenal content remains out of reach to summary by pure information. But this same phenomenal experience also becomes instantaneously unavailable to our own minds the second our perception changes. If I look away from the red object, it seems like the only way I can use that past phenomenal experience is via the information I’ve drawn from it that I can record. “I saw a red object, it has the same wavelength of light as other red objects. Also, I saw a deep redness of red. Aaaaah!” It seems like thought and understanding cannot depend on what we’re phenomenally experiencing at any one time. I can form thoughts about the color red without seeing it in the moment. And when we’re not experiencing something directly, it seems like the only access we have to it is via the third-person information it provided (our past selves are effectively other people here). So unless there are types of thinking that we can only perform as we’re having the phenomenal experience in the moment, it seems to me like the only parts of consciousness that can affect thinking are reducible to access consciousness. That means that for thinking about whether AI models can think, we need to turn to theories of access consciousness. Whether AI has phenomenal consciousness thus looks to me to be pretty unrelated to whether it can think.

Access consciousness’s role in thinking

Consciousness is a controversial topic. A recent attempt to map all theories of consciousness lists over 200. As someone who’s far from an expert myself, I can’t confidently argue for any one. But only a few theories get most of the serious academic attention, and they all contribute to my case that AI can think. I’m going to summarize what each says about the role of consciousness in thought, and how it compares to the folk “movie theater” picture of the mind. In every one, processes we’re not conscious of do the actual work of thinking. Consciousness is at best a way of coordinating those processes, or something separate from them that’s helplessly along for the ride, just justifying their results. A few of the most prominent theories imply that everything we’d call thinking could happen with no consciousness at all.

Global workspace theory

Global workspace theory is probably the most popular scientific theory of consciousness right now. It describes the brain as full of specialized processes all running at the same time, almost all of them subconsciously. Some processes do things like recognize faces, or enforce rules of grammar, or keep you physically balanced. Each one is good at its own narrow job in the broader system, and they mostly cannot communicate with each other directly. But they share a general “workspace” that can only hold a little at a time, so different pieces of information compete to get into it. Whatever wins gets broadcast to all or most of the processes in the brain at once, so the whole brain can use it. According to this theory, being conscious of something is just the information being broadcast.

The theory’s creator, Bernard Baars, ironically used a theater metaphor to describe it. The workspace is the stage, attention is a spotlight on the stage showing what’s on it, and the audience is the giant crowd of subconscious processes in your brain. In this picture there’s no self, and no single you in the audience watching the whole thing. Baars himself said “You don’t have a little self sitting in the theatre.”

Some neuroscientists (most prominently Stanislas Dehaene) have tried to translate this into a theory about physical brain tissue, where the workspace is a network of neurons with long-distance connections. When you become conscious of something, activity suddenly spreads across that network.

If the theory is right, individual mental processes should each have information the others don’t, and we only become conscious of it when it’s broadcast to lots of them at once.

The first part of this claim is pretty self-evident. One of my favorite tests for this is to think of a word at least 5 letters long, and then (without looking!) try to picture where each letter is on a standard QWERTY keyboard. I personally find this is very hard. Next, place your hands on a hard surface like a table, and mime typing out the word as if you had a keyboard below you. When I do this, my fingers show me exactly where each letter is. The muscle-memory part of my brain had this information all along, and wasn’t sharing it with the part that forms visual memories.

Scientists have come up with ways to try to test this theory. In one 2009 study, researchers flashed numbers on a screen too quickly for people to consciously see them. People could still do slightly better than chance at simple operations on the mystery number, like naming it, saying whether it was bigger than 5, or subtracting 2 from it. But they failed at chaining these operations together (like subtracting 2 and then saying whether the result was bigger than 5). This suggests individual subconscious processes picked up bits of information about the number, but couldn’t combine what they had, because the number never got broadcast to the whole brain. A similar pattern will appear in AI models in Part 3.

This theory stands out for giving consciousness a clear, active role. But even here consciousness is more of a spotlight than a director of what’s happening on the stage. It only needs access consciousness, with no transcendent viewer guiding things from above. This also doesn’t rely on the specific nature of neurons, because a machine could just as easily broadcast information to lots of individual processes.

Higher-order theories

Higher-order theories of consciousness say that a mental state is conscious if your brain is also representing itself as being in that state. A red apple might be in your field of vision without you being conscious of it. You only become conscious of it when another part of your brain forms the belief that you’re seeing the red apple. The philosopher David Rosenthal is the best-known defender of higher-order theories.

Blindsight, a bizarre condition, seems to fit this theory well. Some patients with damage to their primary visual cortex report that they can’t see in part of their visual field, but do significantly better than chance when asked to point to things in front of them or guess basic shapes. A higher-order theorist might explain this as their brains still processing some visual information without the higher-order representation required for conscious seeing.

These higher-order stories the brain tells can easily be inaccurate. Worse, they don’t seem to play much role at all in most of what we normally consider our conscious thought and decision-making, like reasoning or planning. Rosenthal argues that reasoning and planning are done by other mental processes whether or not they’re represented in consciousness, and their becoming conscious “adds no significant function.” So under this view, thought and understanding do not seem to depend on consciousness at all.

Attention schema theory

Attention schema theory (AST) describes the brain making lots of important but subconscious decisions about where to spend its limited processing power. Giving a process more of that power is what it means to give it “attention.” If you’re trying to follow one person in a crowded room full of other conversations, your brain can give much more processing power to their words. Michael Graziano, who came up with AST, points out that your brain keeps a simplified model of your body to control your movements (reaching for a cup means making a lot of assumptions about where your hand is). He proposes that it also keeps a simplified model of its own attention, to control where that attention goes. Among other things, it would be bad to let anything that shows up in your senses grab a lot of processing. Your conscious awareness is this simplified model your brain runs to manage itself. Here consciousness plays a useful role, giving other brain processes information they can use to manage how much processing goes to each task.

Introspection under attention schema theory is also a simplified model of what’s happening in our brains. Again, our consciousness doesn’t have direct clear access to what’s happening in our own heads. Your sense of yourself as an indivisible observing entity is actually the brain telling a useful simplified story about itself.

Graziano thinks that AST implies that a machine with a similar attention model of itself could form the same conviction that it has subjective awareness. So this is another theory where access consciousness plays a useful role in the brain that could be replicated in machines.

Predictive processing

Predicting our incredibly complex environment was central to our ancestors’ survival, as it is for other complex animals. Predictive processing approaches make this central to explaining conscious perception and how the brain works. Here, your brain is constantly predicting sensory input using internal models shaped by its experience. Prediction concerns both what is causing your present sensory input and what future input may arrive. The brain compares these predictions to incoming sensory signals and updates its internal models to better match them. On this account, your conscious perception is your brain’s best interpretation of sensory signals. It’s the brain’s best predictive guess about the world. In this theory, you don’t draw raw information into your conscious mind and then interpret it. Your conscious experience is the interpretation of the raw data itself. A lot of our everyday experiences can be explained this way. The reason you usually can’t tickle yourself may be that your brain predicts the sensory consequences of your own movements and dampens its response. Looking into a hollow mask from the back can create the illusion that it’s a face bulging out at you instead, partly because our brains strongly expect faces to protrude outward rather than cave inward.

This is remarkably similar at a broad level to the basic idea behind modern language models, where by “merely predicting the next word” models develop complex and subtle internal representations during training that enable them to write coherently about the world. I’ll flag here though that Anil Seth is a prominent supporter of predictive processing as an explanation for what happens in the brain, and he’s also skeptical that current approaches to AI will produce consciousness. Among other things, he suspects that consciousness is too tied up with the actual biological processes that keep us alive for computation alone to reproduce it. He accepts that machines can be very intelligent and separates consciousness from intelligence. He also entertains the possibility that machines could understand without being conscious, though he remains cautious about whether current language models do. My own take is that this theory sounds strikingly similar at a broad level to descriptions of large language models, to the point that while I accept that they might not be conscious, taking predictive processing seriously as an account of perception would make me more open to thinking that current AI models can think and understand.

Integrated information theory

I’ve always found integrated information theory (IIT) strange, but enough smart people are into it that it deserves a mention, though I’ll note that in 2023 over 100 researchers signed a letter calling it pseudoscience.

IIT’s starting observations include that experience is unified (you experience the parts of your visual field together, as one experience), and specific (you have one exact experience out of countless possibilities). What physical system could have both of these properties? Maybe something with parts that influence each other in such a way that the system as a whole is irreducible. Dividing it into independent parts changes the cause-and-effect structure of the whole thing.

As an example, the system underlying the human visual field can’t, on this view, be like an idealized camera’s sensor array. The array is made up of individual sensors that don’t notice or react when others are on or off. You could divide that array in half without changing how each half works. The brain instead contains dense loops of mutual influence. IIT tries to quantify integrated information in a system’s cause-and-effect structure with a score (called “phi”), which, according to the theory, measures how conscious a system is. Consciousness, on this view, is specific, irreducible cause-and-effect structure within a system.

One interesting fact that lends some credence to this view is the fact that the cerebellum has about 4 times as many neurons as the cerebral cortex, but damages to it seem to not affect consciousness nearly as much as the cerebral cortex. The cerebellum has a lot of relatively independent processing units, which according to IIT give it low phi.

This has a lot of weird implications, that make me think it’s probably not correct. The computer scientist Scott Aaronson pointed out that IIT’s method of calculating phi could make simple logic-gate arrangements that don’t do anything interesting or useful in the world more conscious than humans. The creator of IIT does accept that a sufficiently large grid of logic gates could be conscious even while it’s inactive. That seems pretty bizarre, but consciousness is bizarre, so here we are.

My read is that IIT is specifically about phenomenal consciousness. I’m not sure what role access consciousness plays in the theory, and I can infer that it doesn’t play much at all. As an example, a one-way network without any loops or dependencies has zero phi. And any network with loops can, in principle, be rebuilt as a one-way network that produces identical outputs. This seems to mean that the theory implies that anything and everything done by a conscious entity could also be done by an entity with zero consciousness. A recent paper using IIT as its framework seems to agree that consciousness and all abilities of the brain come apart, and so AI can do everything the human brain does without being conscious. Thus, IIT does not rule out the possibility of thought and understanding in AI systems.

These all seem to imply that the aspects of access conscious important for thought could exist in machines

In all of these theories, most of the work of thinking happens outside consciousness, and where consciousness does help, it’s by sharing information or managing where processing power goes. If that’s consciousness’s whole role in thought, it seems like something we could easily imitate in AI. None of this requires the AI to have phenomenal consciousness, only (sometimes) access consciousness. And if access consciousness is mainly about managing the flow of information (sharing it across a system, and deciding how much processing power different parts of the mind get), that sounds like something a machine can do. So even if you’re very skeptical that machines could ever be conscious, that tells you very little about whether they can think.

Any part of your mind that understands something is made up entirely of parts that don’t understand

This one seems too obvious to need saying, but it helps keep me grounded in the debate, and a lot of people talk as if it isn’t true.

Unless thought and understanding are supernatural and happen outside of physics, any system that understands or thinks is ultimately made up of fundamental particles and forces. These on their own don’t understand or think. Thinking is an emergent property of arranging them in specific ways to perform specific functions. At a higher scale, individual neurons also don’t “think” or “understand.” They are building blocks for a specific system that thinks and understands. All that human neurons do is receive basic chemical signals, change their electrical activity, and release some chemicals. Any system that understands will be made up of parts that “only” do simple things. That’s a basic part of living in physical reality.

It’s surprising how often people will say other systems can’t think or understand because they are made up of individual parts that don’t understand. People will say that an AI is “just doing matrix multiplication,” as if the fact that none of those individual mathematical operations understands anything means the whole system cannot understand anything either.

The role of access consciousness in thought, like anything else in the world, also needs to be completely explained by individual processes that are not conscious. If thinking requires some higher-level awareness, that awareness also has to be explainable in the language of physics (again, unless you believe it happens in a separate nonphysical world that interacts with ours). Dennett once observed (in his book Brainstorms) that explaining an intelligent system means breaking it into smaller, dumber systems, then breaking those into even dumber ones, until you get down to parts so simple that they can be “replaced by a machine.” I think this is the only way minds can ever be explained.

Language, meaning, and consciousness

The folk picture also leads people astray about what it means to understand language. The Cartesian theater picture of language is that to understand a word, you need a clear, first-person experience of what the word refers to. Anything without this experience doesn’t really understand the word. According to this theory, when I say “apple,” you understand me because you can summon a first-person image of an apple, or at least have experienced one before. AI might have a huge collection of associations with “apple,” but without an inner Cartesian theater to experience what the word refers to, it can never truly understand it. So (the theory goes) AI models only manipulate words, instead of using and understanding them the way we do.

Again, I’ll start with some first-person experiments you can run on yourself. Think about the word “dog.” Maybe a picture of a dog appears in your mind. This passes the folk test of understanding. Now think about the word “unless.” What appears in your mind when you do that? You obviously understand “unless” perfectly well. But it’s hard to see what clear inner experience could contain the meaning of “unless.” “Unless” changes the relationships between other things you could imagine, and mental pictures don’t seem much use for a word like that.

So maybe we can understand relational words without mental pictures, but simple nouns do require mental pictures. Already, this seems like a weird arbitrary distinction. If you can fully understand a word like “unless” without a mental picture attached, this suggests mental pictures aren’t that fundamental to understanding. Another problem with this view is that some people have aphantasia and can’t form visual images at all. These people can go most of their lives without even noticing that their mental experience is weird or different in any way. If understanding words required mental images, these people would regularly use language incorrectly, or not really understand the words they use. That seems obviously wrong.

You might object that these people have still seen the things they talk about. They’ve still seen dogs even if they can’t form pictures of them. But there are obvious counterexamples. Do blind people not understand the language they use? Do I not understand the word “Beijing” because I’ve never been there? At this point, I worry that when people say “understand” what they mean is “know everything there is to know about” something. I don’t know everything there is to know about biology, and I can’t form a clear, complete picture of it in my mind, but I can use the word, and I feel like I deeply understand what it means. This understanding is, I find, partly inaccessible to my consciousness. There are a lot of details about what “biology” means that I can’t hold in my conscious mind all at once.

So it seems like phenomenal conscious experience can give us new information about something. It can show me what it’s like to experience the red of a sunset. But a blind person who can’t experience this obviously still “understands” the word “red.” They can use it coherently and know how red relates to lots of different things in the world. They’re just missing one specific aspect of it: the phenomenal first-person experience of it. This seems more like me using the word “Beijing” without having been there than me using the word “Buchladen,” which I don’t know the meaning of at all. Phenomenal consciousness does not seem to be required to completely understand the meaning of the words we use.

This gets at a confusing way we use “understanding” to mean two different things. It can sometimes mean a deep first-person knowledge of what an experience is like. If someone told me that I don’t understand what it’s like to live in Beijing, I’d agree. There’s a lot about the day-to-day experience of living there that I don’t know. But if someone said that I don’t understand the sentence “what it’s like to live in Beijing” that’s obviously wrong. I know what the words are referring to and how to use them. In conversations about AI, these two definitions are sometimes blurred. When I say “AI understands the words I’m saying” I worry that what people think I mean is that AI has some first-person subjective experience where it deeply relates to me. All I actually mean is that AI has enough information about the words, and their relationship to all the other language and images it can process, that it can work with the information in them about as well as I can.

My own view is that understanding a word is a matter of what it lets you do with the word. Consider what it means to understand “fragile.” Someone who really understands “fragile” knows that dropping a fragile object is risky, and that whether something is fragile often depends on what it’s made of. They can use the concept in new situations. To me this looks like having a lot of specific, hard-to-articulate information clustered around a concept like “fragile” (including the logical relationships it implies, like that fragile things are hard to ship), which often can’t be boiled down to a single clear idea we can hold in our heads. Our conscious experience is just one way of getting closer to the right cluster of information and relationships, and it isn’t what holds that cluster together.

One role language plays in thought is letting us understand things we’ve never encountered by connecting them to things we already understand. I can explain a tool you’ve never seen by describing what it does and what it’s made of (all things you already understand), and you can understand the new tool without a clear mental picture of it or any first-hand experience of it.

A problem in the philosophy of language often brought up in discussions about AI is the “grounding problem.” Our language can’t just be a web of relationships between words. It has to be grounded in the reality it describes, so the relationships between words need to be determined by the relationships that actually exist in the world.

This is an interesting problem, but the folk theory often leads people to believe that any solution to the grounding problem requires a first-person conscious observer seeing for themselves “what the language is about.” They need to have the phenomenal experience of the world and connect that to the words they use. If we drop the folk theory, our observations become basically a way of collecting a lot of information about the world at once. As long as information from the world shapes what language says about it, well enough that predictions made in language match what’s actually happening, that seems completely adequate for solving the grounding problem. We don’t need floating observers from outside physical reality giving words meaning based on subjective experiences that information processing can never reach.

The grounding problem does mean that information processing alone isn’t enough for language. A being could have a perfectly coherent internal language that does not at all relate to anything in the world. The main question for whether something “understands” its words is whether they’ve been shaped enough by information from the outside world to reliably describe and predict it. I’ll argue in Part 3 that AI has several ways of getting this information, mainly during training.

So once we drop the folk theory, understanding language means being able to use words correctly by other speakers’ standards, make a huge number of subtle, correct connections between each word and other words, use words shaped by the outside world to correctly describe and predict what happens in it, apply them in unfamiliar situations, and correct mistakes. I think phenomenal consciousness, if it exists, is basically one way humans acquire the information that structures our language, not the thing that gives our words meaning. If something has all of these properties, I would say it understands words for the same reason that humans understand them.

Conclusion

Here’s a summary of the picture I’ve tried to paint here:

• A lot of the work of thinking happens before we become aware of the results. Whatever story we tell about thought and understanding needs to include the fact that a lot of it is capable of happening outside of our conscious attention.

• Our inner world is often opaque to us. It is easy to quickly develop incorrect beliefs about our very recent conscious experience and decision-making.

• Introspection is just another fallible mental activity that draws on information from the world and past mental activity, like anything else. We don’t have some inner god’s eye view of our mind where we can clearly see everything about our own thought process. Perfect self-knowledge can’t be a requirement for AI thinking, because humans can think without it.

• Careful reasoning depends on specific complex kinds of coordination, like holding onto intermediate results, bringing the relevant information together, understanding the semantic content of our words, noticing conflicts, directing attention to different places, and revising mistakes. These are substantial, but none of them currently seem inaccessible to AI models. Without a Cartesian theater guiding them from above, this looks much more accessible to machines.

• It seems like whether AI has phenomenal consciousness does not matter for whether it can think. What matters is whether it has the specific qualities of access consciousness that facilitate thinking. The best theories of access consciousness all seem to imply this is possible in principle. Having an experience and being able to work with all the information that experience would provide seem like different things, and thought and understanding seem to exclusively depend on the second.

• We don’t actually have to settle every debate and disagreement about consciousness before taking machine thinking and understanding seriously, because many of the most popular accounts of both phenomenal and access consciousness imply that AI could think and understand with the right kind of information processing.

• The parts of a system do not need to understand anything for the whole system to understand. Individual neurons don’t understand what’s happening. The argument that a system cannot understand because it’s made of fundamental building blocks that don’t understand is confused. Everything physical is made of these building blocks, so any story of systems that understand has to build them ultimately entirely of things that don’t understand and could be “replaced by machines.”

• Understanding words requires knowing what follows from using them, which is structured by how they relate to the outside world, and their broad logical and semantic connections to all other language. It does not require clear coherent phenomenal conscious mental images.

Once we poke at the folk conception of our minds, the picture that comes out is a mind with abilities spread across many interacting process. Some receive information, or find patterns, or catch mistakes, or generate possibilities to investigate, or follow logical steps from premise to conclusion, or broadcast information to lots of other parts. Our stories we tell about our own minds are products of these imperfect piles of capabilities working together in complex ways, aided by what we’d call access consciousness, which itself appears to be a pile of specific information processing abilities. It’s amazing what these all can do together.

This all I think radically changes what we should expect from machine thought. When investigating whether a machine can think, this gives us way more specific questions to ask, like whether it can represent the world with models, whether it has information about the subtle semantic content of the words it uses, whether it can monitor the output of its past processes and look for any problems or contradictions, whether it can follow a trail of logic from premises to conclusions. Asking “but is it human?” after all this makes less sense to me. It seems like asking whether a submarine is a fish after establishing whether it can dive, navigate, and cross an ocean. These are all individual qualities that make up but don’t completely summarize both fish and submarines, and confusing them as being inseparably tied together in either is obviously silly. We have strong evolutionary reasons to treat the human mind as special, but once we poke at what we actually know these abilities also look like amazing but separable abilities and functions that could with the right technology be replicated in machines.

I haven’t yet given you any reason to think that current AI can do any of this, that’ll be in Part 2 and 3. I hope I’ve shown why “the AI doesn’t have conscious experience” can’t end the discussion about AI thought. By itself, the claim (if it means phenomenal consciousness) gives us very little reason to think that a system cannot do most or even all of what we mean by thinking. I believe that the main question facing us now with AI is whether AI will be able to do all the types of thinking human minds can do, or even valuable forms of thinking we’ve never accessed, and what the world looks like when they can. The difference looks way more like one of degree rather than kind to me.

I worry that people who want to enforce strict bans on “anthropomorphic” language being used to describe AI are often implicitly enforcing the folk conception of how our minds work. They do not want to pick apart the mind’s cognitive abilities and list which AI has and does not have, because for them the human mind seems so obviously special and transcendent that it’s somewhat sacrilegious to imply it could even be divided up into these individual abilities, cutting them off from the unifying light of phenomenal consciousness. This enforcement of a ban on anthropomorphic language is often done in the name of cold hard science up against superstitious suggestible people who have mistakenly identified a ghost in the machine. In reality, I’m worried the people enforcing it have superstitiously discovered a ghost in themselves that isn’t there, and have closed themselves off to a serious attempt to move beyond our own self-concepts and demystify the surprisingly inhuman processes and forces that underly our own minds to the point that some of those processes could be considered to have already been achieved in machines.

How this could all be wrong

I’ve painted a picture here that’s very sympathetic to physicalism, the idea that everything (including consciousness) is physical. Physicalism could be wrong, and that’s the main way I think my story could fail. In polls of professional philosophers of mind, physicalism is more popular but far from a consensus in the field.

It might be that phenomenal consciousness has nonphysical aspects that can’t be reduced to access consciousness, that these aspects somehow depend on the material the mind runs on (brain matter vs. computer chips), and that thinking and understanding depend on them way more than I’m giving them credit for. I think the odds of this are low, but not too low, so I’m open to the idea that machine thinking will be permanently limited by lacking this nonphysical ingredient. But my default assumption is still that this is all wrong, and that machines can think.

The other way I could be wrong is if the material or structure of brains lets them do things with information that computers, given their design, never can. That would mean computers are more limited in how well they can mimic access consciousness. This is interesting, but it doesn’t seem likely to me that computers will be so limited they can’t mimic most of the important stuff in thinking. I think the case that they aren’t gets much stronger once we look at what current AI can actually do, and why, which is where Part 3 will start. Before that, we’ll need to explore the ongoing philosophy debates around whether computers and nonbiological matter in general could possibly think, and that’s where Part 2 will pick up.

In a more legible form (already an imperfect metaphor), this says that all ordinary matter is some combination of these particles (which are really ripples in underlying fields), acting on each other through fundamental forces:

If you put them together in juuuust the right way, you eventually get Antarctica, evolution by natural selection, or tables. These things are our imperfect maps and metaphors for the physical reality underneath, but we often come to believe they’re foundational entities in the world, as if the lamp I see in my room shares some fundamental lampness with all other lamps, and at the heart of reality there are neatly sorted categories like “lamp” or “continent” or “human.” The category of “lamp” is useful, but it would be a mistake to think that lamps are a fundamental part of reality, and that there will be 100% clean and predictable ways all lamps behave that I can infer from any one of them.

A frustration I regularly have in debates about AI is when people act as if the words we use about it are getting at fundamental categories of reality itself, even when they’re clearly especially imperfect, pragmatic metaphors we use to predict our human-scale world. In these moments I kind of want to grab the other person and announce to them that the concepts we use to think about the world in general are just very imperfect social conventions we use to try our best to find these basic but fallible patterns in the incomprehensible complexity lurking beyond our perception. This is obvious in most other circumstances, but it somehow often gets lost in debates about the future of AI.

One example that drives me crazy is the statement that AI is “just a tool.” People seem to infer from this that AI is and always will be completely inert, like a hammer, and that it’s the person using it or designing it who’s responsible for anything bad that happens. This just assumes away the possibility that AI could do things the user doesn’t want, or that the designer didn’t expect and had no way of foreseeing. How do people arrive at this conclusion? By studying how AI actually works? It looks to me that instead they mostly get there because AI, like tools, is something created by humans to be used by humans to solve problems, or that it’s not a literal biological animal. I don’t see anything in how AI models are trained that implies they’ll always behave the way we want, like a hammer does, but this metaphor seems to convince a lot of people that they will. Why?

When people say this, I worry that they believe they’ve accessed some real part of fundamental reality called “tools” and discerned that AI has this same tool-ness as hammers. It seems obvious to me that the word “tool” is a useful but wildly imperfect social game we’re playing to try to make sense of reality. There’s no fundamental tool-ness at the heart of reality. Anything from a rock to a nuclear weapon can, in the right circumstances, be called a “tool.” In some sense a vaccine is a “tool,” but some vaccines contain live pathogens that, in rare cases, behave in ways nobody wants. Does the fact that the vaccine is a “tool” tell us much about how the pathogen inside it will behave? Or that no one could ever have a bad reaction to it unless the person who made or administered it intended that to happen? It seems like the fact that the vaccine shares some properties with other tools doesn’t automatically tell us anything about other unrelated properties it might have. I need some clear additional reason for the fact that a vaccine is a “tool” to imply that it shares any one property with other tools. I can’t just say “Well it’s a tool, so therefore like a hammer it will only cause problems if the person using it intends it to.” That’s obviously silly! Yet in conversations about AI, moves like this regularly fly.

I would think that “AI is just a tool” is meant as a stepping stone to a real debate about whether it’s controllable the way other tools are, but instead most people who say it seem to want to frame the entire conversation around any and all evidence that AI has some inherent property of tool-ness that other things in the universe share. If it’s created by people, that gets counted as evidence that it has this tool-ness, and therefore evidence that it will be inert and controllable. I don’t believe this kind of property actually exists, so this step can’t be taken. It’s all just atoms and the void! So I don’t believe that a fact like “it was created by people” can tell us much about whether AI has some other, unrelated property, like “it will always be easy for the user to control,” just because the completely different objects we call “tools” happen to have both. Drawing this inference seems about as reasonable as saying that because candlesticks aren’t made the way weapons are, a candlestick can never be a weapon. It assumes there’s some part of fundamental reality that literally always connects how a thing was made to what it can do, and I just don’t see the connection.

This has also come up for me in a lot of conversations about when it’s right or wrong to “anthropomorphize” AI. I agree that because AI writes as if it’s human, it’s often tempting, and wrong, to ascribe qualities of the human mind to it that it doesn’t currently have. But critics of anthropomorphizing AI often go way farther and imply that “human” is itself a fundamental category of reality that a system either belongs to or doesn’t, and that because AI isn’t human, it will never be useful to describe it in anthropomorphic terms. This seems completely mistaken to me. I don’t think “human” is a fundamental category of reality either. What we experience as human is the very specific result of incredibly complex systems of biology, neural wiring, and social and economic interaction, each of which is itself made up of simpler nonhuman systems working together in nonhuman ways to give rise to entities like us. To say that we ourselves are human is itself a useful simplification of the complex nonhuman processes that work together to make us up. When I attribute intent to another person, I’m not actually saying “There is some fundamental property of the universe called ‘intent’ that this person is experiencing.” What I’m actually saying is “The complex patterns in this person’s brain and behavior line up in such a way that saying they ‘intend’ to do something helps me predict what they’ll do and experience.” Everything we consider human is a pattern that emerges from simpler nonhuman processes, and the words we use to describe ourselves are imperfect metaphors that help us imperfectly predict and react to those more complex processes in ourselves and others.

Our minds don’t actually work like our folk theories of psychology suggest, and yet our folk language about them still often yields good predictions. Because anthropomorphic language is ultimately a useful fiction imposed on a much more complex process, there are times when it might also be a useful fiction to impose on AI. I think saying an AI “filled out a form” is a useful description of what it did. It often, but not always, yields the same true predictions about the AI’s behavior as it would about a person’s. Similarly, the most recent models check a lot of the boxes in my own definition of what it means to “think.” I don’t think AIs have first-person experience, but they seem to form world models, move step by step through chains of reasoning, and manipulate words in ways that seem impossible without granting that they basically “understand” them. Is it wrong to say that AIs “think”? I think this debate should focus entirely on the pragmatic question of how saying so would affect our predictions about AI. It’s bad to say AI can “think” if it communicates to the other person that the AI is very humanlike, has first-person experience like we do, or has complex emotions and intentions like ours. But it seems useful to say AI can “think” if it communicates that AIs can work with language the way we can and draw reasoned inferences from that language based on world models. In this case, it seems good to anthropomorphize AI for the same reason it’s good to anthropomorphize people: both give us good predictions about the world. In figuring out when it’s right or wrong to use “anthropomorphic” language, it seems useless to frame the question around some abstract fundamental category called “humanness,” existing above and beyond the physical world, that AIs either are completely or not at all tapping into. I don’t think this is something that actually exists, and so trying to figure out whether AI can access it or not won’t tell us anything about what they’re currently like or will be like in the future.

We aren’t able to think without simplified concepts and metaphors. But we have a responsibility to recognize that basically all our concepts are simplified and imperfect. Figuring out the near-term future of AI requires us to work out the specifics of when our imperfect metaphors are useful or not. This debate is not helped at all by conversational games that imply that the speaker has accessed some fundamental category in the universe that AI belongs to along with other entities, so it must always be exactly like those other entities, and any property they have, it must have too. The same game gets played in reverse, where AI is declared to be outside some category like “human,” so it can’t share any property with the things inside it. Neither move tells us anything, because the categories aren’t real. There’s no tool-ness at the bottom of reality that guarantees AI will stay as controllable as a hammer, and there’s no humanness that AI is locked out of that settles whether it can think. Words like “tool,” “human,” and “think” are shorthand we invented to get by in a world far more complicated than we can perceive, and the only real question about any of them is whether applying the word to AI helps us predict what AI will actually do. If it does, we should use it, and if it doesn’t, we should drop it. Whether AI carries within it some fundamental nature of tool-ness or human-ness is a terrible way of approaching a question about it, because like everything else, AI is ultimately just atoms and the void. Or more precisely, the universal wave function.

But AI images and music have turned out to be pretty emotionally resonant with me at a level I didn’t expect, for another reason. They let me curate very specific vibes that matter to me and would be hard to assemble anywhere else. Curation isn't creation, but it's a separate and (I think) equally valuable outlet. A way to get at intuitions that are hard to put into words but feel important to communicate.

My basic experience of creating an AI image is like finding a neat rock. I’m not responsible for the shape of the rock at all, but I might be interested in putting it into an arrangement with other rocks and showing people my specific taste in neat rocks. Maybe I get some points for knowing where to look for the rock, for knowing broadly what types of rocks appear where. Playing with an AI model is like searching its high dimensional vector space for neat rocks, with a taste for specific rocks I can only find in specific places.

Subscribe now

It’s also like making a music playlist. When I put a playlist together, I obviously know I didn’t create the songs themselves, but I’m hoping that by associating with them, and combining and contrasting them in specific ways, I can get across some general sense about the world I think is important. Typing a very specific style prompt into Suno and then playing around with the result feels pretty similar to creating a playlist. I’m venturing out into the vector space of the model to try to find extremely specific sounds I value a lot to add to it. I’ve had way more success here than I expected. The first 2 minutes of this collection of shoegaze songs Suno made for me is a very specific sound I value a lot and want to associate with, especially the big explosion at 50 seconds. It gets across a very specific inarticulable sense of openness to the world I had when I was a teenager. I feel like if I could just glue this sense together with specific images and smells and memories and vibes, people would really “get” me.

No AI music has reached the same level of emotional resonance for me as my favorite bands. I like the sound of the shoegaze songs I made a lot, but the best human-made albums in the genre are in another stratosphere, and I don’t expect AI to approach this quality soon. Suno’s already showing too many signs of being too optimized for more general mid taste. Still, comparative advantage exists. The best human artists remain in a completely separate league from basically all AI music for me, but I don’t always want to listen to the best stuff. Sometimes I want a very specific sound I haven’t heard before in a genre I like, and the best shoegaze album was so good it kind of singlehandedly destroyed the genre, so there’s less human made shoegaze to pick from than I’d like. AI gives me a way of reaching out into the vector space of all possible mid shoegaze to say “This… this is what I’m into, this ‘gets’ me” and the experience of compiling said mid shoegaze in one place is rewarding in the same way making a playlist is rewarding. For even less explored genres I love like dungeon synth, AI can fill the gaps I feel as I explore the very limited but great set of what humans have actually produced in the genre. None of this feels like I’m creating the music, but it does feel satisfying to put these things I’ve found together in specific ways, to know where to look to find things I like, and to use my past great experiences with music to try to express my taste with the new sounds I find.

Last year I posted a big collection of AI images I’d made that I liked. Again, I didn’t feel artistic responsibility for these images. I didn’t feel like my creativity had gone into them. And yet I did feel like I’d been creative in how I used them to express my taste, in the same way making a playlist can involve creativity, and the post resonated, where multiple readers reached out to talk about how our tastes were way more similar than they had expected. This was a fun experience of feeling a bit like discovering far off people secretly in the same tribe, who were moved in the same subtle ways by similar experiences. It was fun to find this together, and the way I found it was purely via what I consider curation, not creation, and that was really valuable on its own!

I really love finding ways other people have curated what they like and exploring them. The sense that other far away people have resonated with specific art or vibes or other media makes the world feel like it has incredible depth. The world can often feel lonely and flat. Maybe the most flat-feeling image I’ve seen is the famous Breezewood PA picture:

The polar opposite of this image for me is a good curated list of things specific people like, either on their personal sites or Goodreads or Letterboxd or music app. These always feel like the world has so much depth, so many points of surprising contact between what we experience alone. I’d made a long list on my personal site of media I like mainly to just imitate other bloggers who made dense explorable curated lists of media they love. I could spend hours poking through these.

The greatest curator on the internet is Piero Scaruffi, who over 30 years has reviewed basically all music on his website that looks basically identical to when he started it in the 90s. It’s my favorite example of curation elevated to a monumental complex achievement:

Scaruffi’s a cognitive scientist who’s focused on AI a lot and among other things also has what I think is the funniest subtitle of any book on AI:

I mutter “Life in the coming age of incredibly stupid machines” once a week, though more and more as cope.

Scaruffi seems to have reviewed most other culture as well, and also publishes long dense essays on things like the history of knowledge, and curates other lists like his favorite waterfalls.

I’d recommend poking around his website for a while, maybe starting with his reviews of his favorite albums.

Scaruffi has a short (and self serving) post on the “Meaning of Art”:

This flips the status of creation and curation: most creation is really just imitation, raw material for curators to build statements about the world.

This seems a little extreme, but consider how many albums you listen to at least in part to be the type of person who listens to them. How often when you listen to music do you feel somewhat ascended from your everyday life because you’re part of the secret community of people who listen to it, the people who know the canon of truly great music? How often do you feel like listening to specific new music will ensure you’re one of the elect?

The curation of rival canons of great music really exploded as the internet took off, and canons reacting to each other to distinguish themselves, and to rise above all others, got faster and faster. For example, if you were on the 4chan music board /mu/ in the early 2010s, you probably liked a lot of these albums:

I know I did! I worry that when I talked about shoegaze earlier, I added a link to Loveless partly to say “Of course I’m allowed to listen to AI shoegaze, because I know about and am deferential to the truly great shoegaze canon, I’m not some pleb.”

But the existence of these canons encouraged rival canons to pop up, where you could not only feel more “in the know” and ascended compared to most people, but even more in the know by people who thought they were in the know. Take this other list someone on 4chan made, and especially the note at the bottom.

Yerself is Steam is a great album, but in my opinion it’s clearly not nearly as good as Loveless, which the author probably considers /mu/core and “radio music.” This is basically a hostile act of curation, using the art in the way Scaruffi describes to say something about the values and ideology of the critic, separating the critic from rival canons. This becomes more and more far removed from the art itself and is instead part of a drawn out contest of curation.

Once you start paying attention to what curators are trying to do, and how often any specific art is more of a stepping stone to say “I am with the cool good authentic community of people, by listening to my curated canon you will ascend to them too” it’s hard to look away. So many Pitchfork reviews were much more about the type of people listening than the music itself. Look at their 10/10 review of Kid A, how much of it is referencing other key parts of the canon, or the interesting location of the reviewer. Or their 0/10 review of Bachman Turner Overdrive’s greatest hits, which very specifically says “If you like this music, you are not one of the elect”:

Those who still cling to BTO-- and you know who you are-- are usually dorky minivan types who think rock and roll is about having your friends over for beer and barbeque. Let me say on behalf of the world's young people that all you people truly suck. Even in your retro phase there's nothing remotely charming or rocking about you. Here's hoping that next T-Bone you eat leaves you choking on your own excess. Die! Die! Die!

Seems a little much…

This is all to say that curation on its own can be a monumental and impressive task, that it’s incredibly fraught and bound up with our self conception and rival systems of status, but also at its best can be a unique satisfying bridge between people, all separate from any act of creation of the art itself. The idea that AI radically expands the space of curation and opens up ways for individual people to be much more specific with the images and sounds they collect and display to the world as “what they’re about” is under discussed in the conversation about AI and art. Most AI art conversations I see online are still reduced to whether AI art “involves creativity.” For me it involves as much creativity as assembling a playlist. Whether that’s like artistic creativity seems less interesting to me than what it means about the future of how people share their taste with each other, and use it to either find or compete with each other. How much of the good feelings we get when we find others who like the same art is from authentic shared experience vs just the camaraderie of identifying other members of our ascended rival tribes is a pretty foundational question about human life I don’t have a clear answer to. I worry that a lot of people in the AI and art debate fail to appreciate how interesting curation is as a separate concept, and the heights of achievement and depths of emotional turmoil it can reach.

I’ll close with a few AI images I made recently:

Some Massachusetts triple decker art:

Nostalgic photographs:

Geodesic domes:

Subscribe now

I’ve noticed that in lots of conversations about current AI models and the general AI buildout, it’s becoming more and more popular to say that AI isn’t happening "in a democratic way. There are times when I strongly agree. I’m very worried about the ways AI could enable the concentration of power, or impose huge (including existential) risks that citizens never signed up for and would hit the brakes on if they could.

But there’s a specific way of arguing that AI isn’t “democratic” that’s gaining traction, and that I find ominous and antithetical to what makes a good democratic society. It’s usually dressed in the language of progressive politics, but underneath, I think it carries authoritarian, homogenizing and anti-democratic implications.

Subscribe now

Pluralist vs homogenizing democracy

There are two visions of democracy hiding behind our talk about it, one I think is good and the other I think is bad, and you can probably tell which is which by the labels I give them.

The goal of pluralist democracy is to give people with wildly different beliefs about the good life and what’s really valuable control and veto points over how they’re governed, without letting them steamroll each other into all living under one contentious ethical or religious vision of the world.

Most liberal democracies like the US manage to balance pluralism with majority rule. Despite being a majority Christian country, and that majority having voting power over who represents them, the US government is not able to impose Christianity on everyone else. People have a lot of democratic control over contentious issues of government, but they often can’t use this to steamroll other people into very specific visions of the good life. Christians are restrained by governing norms from forcing non-Christians from attending church.

The goal of homogenizing democracy is to discover the true, good ethical values and beliefs about the world, with the faith that the majority either publicly or secretly believes them, and then to structure all society around those true, good ethical beliefs, and push ideological minorities into going along.

Here’s a more ominous way to frame it: “There is an authentic community of rooted, powerless, good people, who know what’s good and right in life and agree on it. They’re up against unrooted powerful people who don’t know what’s good in life and want to override it for their own gain. The good people are in the majority. Democracy means discovering these good values all good people hold, and then representing and acting on it, steamrolling the bad powerful people into going along with these good values.”

I read this as ominous because it’s an especially useful cudgel for forcing people into your specific vision of ethics and the good life. If you invalidate another value system by saying it’s not what all the good, powerless people believe, this gives you more permission to steamroll the people who hold that value system in the name of resisting power. Many Christian nationalists will frame themselves as representing the good, simple, rooted majority beset by powerful cabals who don’t understand their values, and use this to argue that underneath these differences is only a raw struggle for power that the good people need to win. Thus to them, seizing the government to impose Christianity on religious minorities is actually a form of democracy. It’s simply letting the values of the good majority be represented in how the country is governed. Stuff like this is why I worry about accepting a narrative that lets homogenizing democracy sneak in unexamined. Democracy at its best weighs the pluralistic desires of radically different people. At its worst it excludes those desires from the debate entirely, on the grounds that anyone who disagrees with the majority is illegitimate and not worth making concessions for.

And it can creep in under the guise of noble-sounding rules, like “a town should have control over how its resources are used.” For example, consider the claim that a small town should have complete democratic control over what gets built there. This sounds nice in the abstract, but is actually a way for homogenizing democracy to slip in. Imagine what happens if 80% of the town is Catholic, and 10% is Muslim. The Muslim population gets together and says they would like to buy some of the town’s land to build a mosque. Will this mosque get built if it’s voted on by the town? Well, 80% of the town believes that the mosque is part of a false religion that sends people to hell. It seems unlikely that the mosque will get built. By giving citizens democratic control over everything that gets built in their town, we’ve allowed them to steamroll pluralist values and homogenize the area into the contentious beliefs of the majority of people who happen to live there. This is bad!

Some people imply that in a “truly” democratic society, all value disagreement would ultimately be dissolved through deliberation. Communities would come together and deliberate until they discovered the most virtuous or liberatory way to live. I think deliberation like this is a great model for specific decisions the government makes, but it shouldn’t be allowed to seep as far into everyday life, because taken to the extreme it implies everyone must eventually be made to agree on every contentious value.

So while I think that at its best, liberal democracy is the transcendent crowning achievement of human history, talk about democracy can also be used to justify authoritarian homogenizing impulses and needs to be carefully examined. When people invoke democracy, they’re usually smuggling in general rules about how society should be governed, and those rules’ political implications need to be drawn out. The specific model of democracy where the goal is always to discover the good values that need to be imposed on everyone, that all good people either publicly or secretly believe (or would believe if they could be led to the light), and that anyone who disagrees with is not worth considering in deliberation, is a recipe for destroying the delicate balance that allows people with wildly different beliefs to live together and change their values freely.

People in democratic societies have strong reasons to try to frame their opponents’ opinions as outside the sphere of legitimate democratic consideration. An effective way to do this is to frame their opponents as representing “the opinions of the powerful”meaning illegitimate, out-of-touch views trying to undermine the good people. This is often a way of trying to banish different views from the process of democratic deliberation. I’m worried that this framing comes up a lot in the AI debate, used by people who think they’re behaving as the defenders of democracy, but are often actually stifling authentic difficult democratic deliberation between radically different values.

Which visions of democratic AI are pluralist vs homogenizing?

I think AI poses a unique technological threat to pluralist democracy. Society currently exists in a delicate, kind of miraculous balance where most people are able to add a lot of economic value, and have lots of ways to defect from important systems if their political power is threatened. We are deeply interdependent. AI could, under some models, erode our economic interdependence and leave powerful people with less incentive to respect the will of everyday voters. It’s very possible that democracy is a lucky passing phase in human history, built around a specific economic regime that’s existed since the Industrial Revolution that might go away if we enter a new and different mode of economic production. See here for more.

AI also poses lots of other risks, either indirect risks that people will use it to create dangerous powerful new weapons, or more speculative direct risks of AI itself overriding what we want. Just like I think everyday people should have democratic control over whether their neighbors have access to nuclear weapons, I think people have a democratic right to expect the general flow of technological progress to not descend into a specific basin where society enters an incredibly unstable equilibrium of extremely powerful easily accessible weapons.

The dynamics of automation with AI also threaten to concentrate wealth and power in the labs themselves. OpenAI and Anthropic are now some of the fastest growing companies in history. Many of their members have become billionaires, and some local political fights are becoming proxy battles between them over rival visions of AI safety and governance policy. If you expect AI capabilities to continue to progress, this poses a danger of a unique form of corporate power consolidation we haven’t seen in America in a long time, if ever. This also threatens political pluralism, and people have a right to expect that their representatives respond to this new threat. How? I don’t really know, but I’m willing to entertain lots of different answers from libertarians or socialists or anyone in between. I worry our learned political impulses might be as useful in the new era of AI as Aristotle would have been in the Industrial Revolution.

Lots of concerns like this I think are legitimate threats to pluralist democracy, and people have the right to democratically demand forms of intervention that I wouldn’t normally consider for more normal technology. More broadly, a new era of intense automation will demand democratic deliberation between people with radically different values on the shape of the new world that they want to live in together.

I think AI will over the next ten years demand an “all hands on deck” response from political philosophers, economists, and everyday people, and authentic pluralist democratic deliberation needs to happen.

While I’ve been happy with some aspects of how the left’s thinking has evolved on this, there are specific strands of intensely anti-democratic thinking that seem to be attempting to take over the debate and define what democratic AI even means. Here I worry a lot that people are using the language of democracy to push for homogenization and authoritarianism.

A clear example comes up a lot in the data center buildout. A very common line among people claiming to represent democracy in the debate is to say “No one is asking for these! They only benefit billionaires!” I think if I were able to separate these people from the debate, take them out for some nice tea somewhere, have a long conversation with them about something else, and then politely ask “Just being real, do you think there are a decent number of everyday people who think about AI in a different way than you whose demand for more is incentivizing the data center buildout?” I think they would say yes, but that these people are mistaken. We could go on and have a nice conversation from there. I worry that the initial claim that “no one is asking for this!” is basically a way of pushing the vast numbers of people who are in fact asking for this out of the difficult process of deliberation that should happen with any new technology.

There are a lot of people who want to use AI. Chatbots are consistently the most downloaded apps. Half of all web developers polled say they use AI daily. About a quarter of Americans say they use chatbots daily.

More people say AI helps than hurts most things they use it for.

However, Americans also believe AI is more likely to make society worse.

So what do we do with this? How should we think about this from a democratic perspective?

If a quarter of people like a product, two quarters hate it, and one quarter is indifferent, should that product be allowed to exist?

Homogenizing democracy would say no. The plurality has spoken. People with different opinions from the plurality should be steamrolled.

This is obviously wrong, yet I see people imply basically this in lots of debates about data centers. People will often say that people using AI who like it are both a small minority, and so out of touch with what’s good and right that their AI use can’t be allowed to use any resources at all. It can’t be allowed a drop of water or an acre of land or to emit any CO2. When I argue that people who like chatbots need to have their own demands represented in the market and in democratic deliberation, I’m often met with a knowing sneer, like “Oh, this guy hasn’t gotten the memo. These people using AI aren’t really ‘the people.’ They’re alien outsiders and don’t get access to the sphere of deliberation. Real democracy means making sure these people are completely disempowered and are never able to use any community’s resources.” To anyone who actually thinks about political philosophy at all, this is a bizarre and sad mangling of the democratic impulse. It’s an intense assertion of anti-democratic power for one specific faction in the debate. For any other unpopular product, we’d recognize how ridiculous this is.

One way to steamroll behavior you don’t like is to argue that it imposes lots of externalities on other people, so it’s actually an attack on their own freedom. Despite data centers not really imposing more externalities on locals than any other normal industry (from what I can tell), a huge media bubble has developed where wild stories are told about data centers forcing whole towns to evacuate, of infrasound ruining people’s lives, of water polluted and taps running dry, most of which evaporate the moment you poke at them. People seem to have a strong desire for these stories to be true. There was a recent New York Times opinion column on how the data center backlash may be a “winning issue for Democrats,” framed almost entirely around the sense of community people feel in anti-data center protests rather than any solid issues with them. I worry a lot of this is a sublimated authoritarian lurch to control the behavior of people doing something you don’t like. Stories about data centers being bad feels good because it hands you a cudgel you can use on people you think are out-of-touch losers. There’s a lot happening here, but I don’t see this as a good sign for authentic democratic deliberation about the future of AI. It seems much more like an impulse to homogenize everyone into a specific way of thinking about AI that they wouldn’t assent to on their own.

This also shows up in very confident claims that AI cannot, by definition, ever produce anything of value at all. At some point this looks a lot more like ignoring the vast numbers of intelligent people saying they’re getting something of value by using it. Critics don’t have to agree with them, but the confident metaphysical claims about the inherent value of AI output start to look more like a lurch to stifle debate by casting people who disagree out beyond the realm of legitimate conversation.

Separate from this debate, another instance of the homogenizing impulse is the idea that individual models themselves should be made to more democratically represent the opinions of the majority. There’s a lot of talk about “for whom” chatbots are designed, implying that they have lurking biases that would be fixed by making them more democratic. I don’t doubt that chatbots (like anything) carry lots of blindspots and misunderstandings of the experience of people very different from the people who designed them and who wrote the data they trained on, but I don’t at all think democracy specifically will make this problem better, and this implication that private products should all represent the democratic will of the good people I think is another way of smuggling in the homogenizing impulse.

Current AI models would become way more conservative if their values were voted on

In my 20’s I went through a huge Civil War phase. Anyone who does the deep dive comes away kind of shocked at the state of public discourse about the Confederacy. While public understanding has improved, I think a lot of people still really don’t know just how bad slavery was or how central of a motivation it was for the South in the war. This is reflected in public polling. More Americans have a positive than negative opinion of Jefferson Davis, for example:

Maybe this is an artifact of most people just not knowing who he was, or confusing him with Thomas Jefferson. Regardless, this is a pretty clear case where I’m very happy that the main AI models deviate from the winning view of Jefferson Davis among the public.

Claude:

ChatGPT:

Gemini:

Maybe someone could program an AI model that more accurately represents the views of the median American. This AI would be unsure about evolution, and in general much more conservative across the board than the average person demanding that AI’s democratically represent the beliefs everyday people hold.

Someone who holds a simplistic, homogenizing view of democracy often needs to invent a story where everyday people all secretly agree with them. If you imagine that the good rooted majority of people all know your values are correct, this gives you grounds to steamroll the few out of touch people who disagree with you. Because they’re out of touch, their divergent views have no claim to political respect or protection.

But the views of everyday people are complicated and often way out of line with progressive visions of the world. There are many values I share with my fellow coastal elite city dwellers that I’m happy to see reflected in AI models. These values show up less outside of cities, and even less in other countries. They are often not the views of either the American or global majority.

I think a lot about this exchange from this 80,000 Hours interview with Mustafa Suleyman, the co-founder of DeepMind:

Rob Wiblin: Yeah. While you were involved with DeepMind and Google, you tried to get a broader range of people involved in decision making on AI, at least inasmuch as it affected broader society. But in the book you describe how those efforts more or less came to naught. How high a priority is solving that problem relative to the other challenges that you talk about in the book?

Mustafa Suleyman: It’s a good question. I honestly spent a huge amount of my time over the 10 years that I was at DeepMind trying to put more external oversight as a core function of governance in the way that we build these technologies. And it was a pretty painful exercise. Naturally, power doesn’t want that. And although I think Google is sort of well-intentioned, it still functions as a kind of traditional bureaucracy.

Unfortunately, when we set up the Google ethics board, it was really in a climate when cancel culture was at its absolute peak. And our view was that we would basically have these nine independent members that, although they didn’t have legal powers to block a technology or to investigate beyond their scope, and they were dependent on what we, as Google DeepMind, showed them, it still was a significant step to providing external oversight on sensitive technologies that we were developing.

But I think some people on Twitter and elsewhere felt that because we had appointed a conservative, the president of the Heritage Foundation, and she had made some transphobic and homophobic remarks in the past, quite serious ones, that meant that she should be cancelled, and she should be withdrawn from the board. And so within a few days of announcing it, people started campaigning on university campuses to force other people to step down from the board, because their presence on the board was complicit and implied that they condoned her views and stuff like this.

And I just think that was a complete travesty, and really upsetting because we’d spent two years trying to get this board going, and it was a first step towards real outside scrutiny over very sensitive technologies that were being developed. And unfortunately, it all ended within a week, as three members of the nine stood down, and then eventually she stood down, and then we lost half the board in a week and it was just completely untenable. And then the company turned around and were like, “Why are we messing around with this? This is a waste of time.”

Rob Wiblin: “What a pain in the butt.”

Mustafa Suleyman: “Why would we bother? What a pain in the ass.”

Rob Wiblin: It was a very striking story to me, reading that in the book. People complain, I think correctly, that decisions of enormous global importance, historical importance, are potentially going to be made inside these AI labs — and the kinds of people who work at these labs are a very small fraction of the people in the world in terms of their political views, the values that they have, the things that they’ve studied, the kind of information that they happen to know. So it would be good if we could get a wider cross-section of the human population involved in scrutinising or having some input on these.

But to share power with the general public in the Global South, or even just outside big cities in the US or UK, will inevitably involve giving influence to people with views that I imagine are very offensive to Google staff — probably more offensive than Kay Coles James, who had to resign from the board for having more conservative, traditional views on gender. So based on that experience, it just seems like it’s very unlikely to happen, when we might just be flat out trying to get acceptance for having more input from a broader cross-section of educated people in the UK into Google. Like, that’s going to be the most that people will tolerate.

Mustafa Suleyman: Yeah, totally. This is part of the problem, right? I mean, 40% of people in the US believe that trans rights are moving too quickly; 30% believe that abortion should be made illegal; 30-odd% are against gay marriage.

Rob Wiblin: And then think globally, right?

Mustafa Suleyman: Right, exactly, then think globally. So I think we have to just learn to sit down with people who we fundamentally disagree with. That goes for China and the Taliban and all these people who hold these views — because if we can’t do that, then we’ve really got no chance of actually hearing one another out and changing one another’s views and making progress. I think in the last two or three years, I feel like we’ve really taken a few backward steps in that direction, and it’s super problematic because it just demonizes the other and then we just end up hating on one another.

And it’s very frustrating for us, because we put in a huge amount of effort to make that happen. Before that, when we were acquired, we made it a condition of the acquisition that we have an ethics and safety board. That in itself was a first step towards this kind of broader public effort. Then after the ethics and safety board, we actually tried to spin DeepMind out as a global interest company: one that was legally governed by the requirement to consider all of the stakeholders when making decisions. So it was a company limited by guarantee. And then the charter definition had an ethics and safety mission for AGI development; we actually had the ability to spend vast amounts of our income on scientific and social mission.

So it was a really creative and experimental structure. But when Alphabet saw what happened with that board, they basically just got cold feet. That was the bottom line. They saw what happened there and they were just like, “This is totally nuts. The same thing’s going to happen for your global interest company. Why do that?” And then eventually we pulled DeepMind into Google, and in a way DeepMind was never independent — and isn’t independent now; obviously now it’s completely part of Google.

Rob Wiblin: Yeah. The core of trying to address representativeness is that you will be ceding power to people who don’t share your values, and if people are not willing to make that compromise, then it’s not going to happen.

I’ll just flat out say that I have zero interest in sitting with what the Taliban has to say about what values current AI models should reflect. The Taliban can just not use the big chatbots, or build their own more in line with their values. I don’t feel any sense of democratic injustice at all that their beliefs are not influencing the behavior of a private product. The only worlds where the opinions of the Taliban become relevant at all for AI are the really wild speculative situations where AI comes to determine basically everything that happens on Earth. I do see some risk of that happening, but few of the people worried that AI doesn’t represent everyone on Earth equally seem to share this particular worry. If AI rises to run all society, I will circle back with the Taliban.

I’m sure the AI models get plenty of important facts wrong about the lives and experiences of people in very different situations from their designers. But the solution here doesn’t at all look like making AI more representative of the average person in the world, because the average person also carries horrible biases against lots of other people. One way to fix this could maybe be selecting specific expert representatives of different under-represented communities or ways of living and having those experts impose specific ways of responding on the AI. Maybe different worldviews could make their own LLMs, each representing those specific views. But I worry that making these private products “more democratic and representative of everyday people” when they already hold more liberal views than most people do would mostly just lead to more wrong or hateful models.

Lurking under this idea that AI models themselves should be “more democratic” is sometimes the idea that all products in society should reflect the correct opinions of the good majority, and never reflect the opinions of the bad minority. It’s again a form of homogenization. I think a better solution here would be for radically different ideological communities to each have powerful AI models of their own, none claiming to represent the democratic will of all good people, something impossible in a complex pluralist world. But I also am just not especially bothered that AI models don’t reflect a lot of the very conservative values of everyday people that I think are just incorrect. I’m not eager for CreationistGPT.

Conclusion

I think authentic democratic deliberation about AI is going to be deathly important over the next ten years, as AI threatens to concentrate economic power and unleash new risks on the world that we’re not ready for. I don’t want this debate to lead to anyone using the banner of democracy to override actual democratic deliberation. We need to acknowledge that there’s already both a lot of demand for and a lot of hatred of AI, and welcome both as full participants in the debate about the future. We should be wary of anyone declaring that barring huge numbers of people from the debate is a form of “democratic resistance to AI.” This threatens to mostly burn the commons of the debate and alienate people from more subtle ways of resisting and democratically governing the unique risks AI poses.

Subscribe now

This post will be a very detailed overview for autonomous vehicle advocates to understand the situation in DC and pathways to legalization.

This post won’t have arguments for why we should legalize Waymo. I have a separate post on that, though I’d like to write an update to it since I think there are much stronger cases I can make.

You may have noticed that Waymos are already driving around DC (with humans at the wheel). They’ve been mapping the city and say they’re nearly ready to move forward, but the city currently has no legal path to commercial service. Two decisions are in the way: a driverless testing pathway and a commercial operating framework. Right now both are held up by a missing District Department of Transportation (DDOT) report that was supposed to arrive in October 2022, and by DDOT’s failure to open a permit pathway separate from the report that already exists in statute.

Subscribe now

Contents

• What’s blocking Waymo

  • The 2 things Waymo needs

  • The DDOT report and the separate testing permit framework

  • The council and mayor debate once the report is published

• The mayoral race

  • Kenyan McDuffie

  • Janeese Lewis George

  • The mayoral race makes the DDOT timeline much more important, and vice versa

• The federal government

• How to legalize Waymo in DC

  • The most promising levers to pull

  • Lessons from other cities

What’s blocking Waymo

The 2 things Waymo needs

Waymo needs two things from DC’s city government that it doesn’t have.

• Driverless testing permits. Right now, every Waymo in DC is required to have a human safety driver. But Waymos need to be tested without a driver. This has been a standard step in every other city. DC law already authorizes DDOT to run an AV testing permit program that can include remote-operator (driverless) testing, and 2025 amendments specify that driverless testing requires that permit. But DDOT has never made the application for driverless permits available. Until DDOT opens the permit pathway and finishes the associated rules, testing can only continue with a human physically present, and the necessary driverless tests can’t happen.

• A commercial operating framework. Even if Waymo got driverless testing permits tomorrow, current DC law has no structure for a commercial autonomous ride-hailing service. Things like insurance requirements, licensing structures, and rules for how a driverless vehicle interacts with police don’t exist. Autonomous ride-hailing is effectively illegal in DC.

For Waymo to operate normally, they need to get driverless testing permits, then begin commercial operations under a legal framework the council has passed. Neither step is available today.

The DDOT report and the separate testing permit framework

The Autonomous Vehicles Testing Program Amendment Act of 2020 gave DDOT authority over AVs and required the department to produce a report within one year of the law's applicability date of October 1, 2021, meaning the deadline was October 1, 2022. The report was to provide “recommendations to safely accommodate the deployment of autonomous vehicles on public roadways for commercial, personal, and any other use.” The report could optionally include draft legislation or regulations.

This report is more than three and a half years late, and is now the main political obstacle blocking Waymo.

The 2020 law tasked DDOT with setting up a formal AV testing program as a distinct obligation from the deployment report. DDOT hasn't indicated they're working on permits separately, so we may not get clarity on testing permits until the report comes out, if it addresses them at all.

It blocks legislation because Charles Allen, the DC Council’s transportation committee chair, has made the finished report a precondition for holding hearings on any AV bill. Former councilmember Kenyan McDuffie (one of the two leading candidates for mayor) introduced a comprehensive AV legalization bill (the Autonomous Vehicles Amendment Act, B26-0323) in July 2025. It’s been sitting in Allen’s committee ever since without a hearing, because Allen won’t schedule one until the report is finished. Nothing in the law says the council can’t legislate on AVs without this report. Allen’s position is a political choice.

The history of the report

• December 2020: Autonomous Vehicles Testing Program Amendment Act of 2020, giving DDOT authority over AVs and required the initial report. The entire mandate says:

  • “Within one year after the applicability date...the Department shall transmit to the Council a report that provides recommendations to safely accommodate the deployment of autonomous vehicles on public roadways for commercial, personal, and any other use the Department determines. The report may include draft legislation or regulations.”

• January 2022: DDOT publishes a notice of proposed rulemaking in the DC Register to create an AV testing program. This never gets finalized. This is a separate obligation from the report, more on this below.

• Late 2022: The report’s deadline of October 1, 2022 passes with no public explanation. I can find no reporting on what, if anything, DDOT was doing on the report during this period.

• 2023: No report, no public explanation for the delay. There’s no reporting anywhere I can find on the reason it was delayed.

• July 2024: A Wayback Machine snapshot of the mayor’s AV Working Group’s webpage (which has since been removed from the DMOI website) reveals that DDOT was still in the procurement phase for hiring a consultant to help define a regulatory framework. More than two years after the report was due, the work of writing it had not yet been contracted out.

• March 2025: Congress freezes nearly $1 billion in DC’s local funds. Allen later attributes part of the report’s delay to this, but the report was already almost three years late at this point.

• Spring 2025: Some expected the report to be finished around this time. It did not materialize.

• Fall 2025: DDOT told Allen the report would be ready around this time. In September, DDOT backtracked, telling Allen it would be out “in the new year, timeline unknown.” Allen said he had been meeting with trial lawyers, safety advocates, and Waymo in preparation.

• Late 2025: DDOT said the report was one of the programs affected by budget cuts. But this doesn’t explain why it wasn’t finished in 2023 or 2024, before the cuts happened. DC has since entered a new fiscal year and the funds have been restored.

• February 2026: A DDOT spokesperson told Axios the agency expects to propose recommendations “later this year.” Waymo told Axios that DDOT informed the company the report “is being actively worked on, and could be finished by this summer.”

Will the report be finished by summer? I’m skeptical. But there are three reasons to think this time could be different. Political pressure is higher than it’s ever been. The Washington Post editorial board has now published two pieces on this (one literally accusing DDOT of “sabotage”), Waymo is spending a lot more on lobbying, and the company got 1,500 residents to contact city hall. Also, DDOT appears to have finally contracted outside consultants to help write it. And the fact that DDOT gave a specific timeframe to both Axios and Waymo, rather than the usual vague non-answers, suggests at least something is happening.

What happened with that AV testing program?

DDOT published proposed rules in January 2022 to create the AV testing program. This is separate from the report. The 2020 law created two distinct obligations: a testing permit program and a deployment report. The proposed rules would have established how companies apply for testing permits and what the testing framework looks like. If DDOT had finalized them on any reasonable timeline, Waymo could have applied for driverless testing permits years ago.

DDOT never finalized them. The mayor’s now apparently dormant AV Working Group’s own page, as late as mid-2024, said the rulemaking was “still in the works” and they looked “forward to keeping the public informed when there is more to share.”

This is in my opinion more ridiculous than the delayed report. DDOT got far enough to publish draft rules for driverless permits and then just stopped. They should really be held accountable here. Waymo could begin driverless testing with permits while they wait for legislation to be fully legalized if DDOT could just get its act together on this separate thing, but instead they’ve gone completely silent on it.

This created an obvious problem when Waymo and other companies started showing up to test in early 2024. There was no permit system, so there were no formal rules. The council responded to this problem with the Autonomous Vehicle Testing Permit Requirement Amendment Act of 2024 (B25-0710), which Allen’s committee held a hearing on in June 2024. The law was simple: until DDOT’s permit program actually exists, driverless testing is prohibited. Any company testing AVs must have a human in the vehicle and give DDOT advance notice.

Because temporary legislation in DC expires and must be renewed, this had to be passed over and over, as an emergency measure in March 2024, a temporary law in June 2024, another temporary law in March 2025, and a congressional review emergency act in March, before finally being made permanent through D.C. Law 25-312 in the same month. DDOT’s inaction forced the council into a patchwork of reactive legislation instead of operating under the coherent framework that was supposed to exist by 2022.

Even if the deployment report appeared tomorrow, there would still be no finalized testing permit framework from DDOT. The report and the testing rules are separate obligations, and both are unfinished. Anyone who tells you the report is the only thing blocking Waymo is missing half the story. This is something I think a lot of AV advocates are missing in conversations about Waymo in DC.

It’s worth noting that the report could end up addressing testing permits. The mandate says the report “may include draft legislation or regulations,” so DDOT could bundle everything together. But there are reasons to think it won’t:

• The 2020 law created them as separate obligations. The testing permit program was established under a different section of the law than the deployment report. DDOT published proposed rules for the testing program in January 2022, a completely separate regulatory track, and then abandoned them. The fact that DDOT initiated a standalone rulemaking for testing permits confirms that the agency itself originally treated these as distinct workstreams.

• DDOT has never said the report will include testing permits. Despite nearly four years of delay, DDOT hasn’t publicly indicated that the report will address the testing permit framework. If they were planning to combine the two, you’d expect them to say so, especially given how much scrutiny this issue is getting.

• The report and the permits are controlled by different bottlenecks. Allen’s precondition is about legislation: he won’t hold hearings on AV bills until the report is done. But testing permits are an executive function. DDOT could issue them under existing statutory authority without any new legislation passing. These are different problems held up by different branches of government, which makes it less likely that a single document resolves both.

What the report might say

The DDOT report’s content matters a lot. It’s possible that the report recommends heavy restrictions, and that would obviously be terrible for the prospects for AVs in the city anytime soon. There are maybe three possibilities:

• A permissive framework would recommend clear pathways for testing and commercial AVs with standard requirements. This is basically what McDuffie’s bill already proposes. It would give Allen what he says he needs, and things could move pretty quickly.

• A cautious, phased framework might recommend a limited testing period, geographic restrictions, fleet size caps, mandatory human oversight periods, and extensive reporting requirements before any expansion. This would still move things forward but could add a year or more.

• A restrictive or inconclusive framework is the worst case. If the report emphasizes unresolved safety questions, calls for further study on labor impacts, or recommends conditions Waymo can’t practically meet (like requiring a human in every vehicle), it gives official cover to delay indefinitely.

A couple things would really help AV advocates: if the report addresses testing permits alongside deployment (rather than leaving those stuck in a separate bureaucratic track), and if it includes draft legislation (as the mandate allows), which could speed up the council’s work significantly.

The Three Gatekeepers

Three parties share responsibility for this block: DDOT, Charles Allen, and Mayor Bowser.

DDOT

DDOT is more than three and a half years late on the report. The agency has a lot on its plate, and when it received this assignment AVs were much more speculative.

Some people have speculated about internal opposition to AVs at DDOT, but I don’t see strong evidence for this. DDOT Director Sharon Kershbaum hasn’t publicly commented on AVs. Some have inferred anti-AV sentiment from her other unrelated positions that they read as anti progress, but I think that’s a stretch. Transit decisions are complicated and don’t map neatly onto a pro- or anti-tech progress axis. I’d identify as pretty pro-progress, but I share Kershbaum’s lack of enthusiasm for dedicated bike lanes (sorry friends), and I don’t think that means I’m anti-progress on other transit issues.

Stephanie Dock is the manager of DDOT’s Autonomous Vehicles Program and leads the team responsible for overseeing autonomous vehicle testing. She was on a panel on AVs presented by MIT in 2025 and had some useful quotes on the report:

We want to be adaptable, we want to be responsive to what’s going, and the industry is changing a lot faster than our rulemaking structure is allowing us to keep pace.

In order for us to do a permitting program, we have to have done a rulemaking.

Trying to figure it out before it comes. Anybody else who’s struggling with that prospect, you are not alone. We are also trying to figure that out.

These are notable because they’re the closest thing to a public explanation for why DDOT has been slow. She’s essentially saying the bureaucratic process itself (rulemaking requirements) is a structural bottleneck, not just a prioritization failure.

On the deployment report:

A deployment report with recommendations for legislation and regulation... that has to be gone through a lot of signatures, very similar regulations, so I have no promise on when the full report will be released, but I... fingers crossed, we’ll have made it through the procurement process so that... such that we can release the research aspects, kind of the background work.

This confirms the report was still going through procurement/signatures as of this talk in 2025, and that she herself couldn’t promise a timeline.

On industry vs. DDOT’s timeline:

Industry has slightly different plans. I am still looking at recommendations for deployment. Industry is certainly saying we would like to deploy. So they said they’re coming in 2026. I am eagerly awaiting how they’re going to figure out to do that.

But they also have political levers that they can pull that I, as a lowly staffer, do not.

So here it seems like she’s openly acknowledging that Waymo’s 2026 deployment plan doesn’t align with where DDOT actually is, and that the company has political tools (lobbying, public pressure) that she doesn’t.

On DC’s attractiveness and risk for AV companies:

I think everyone knows that the risks of coming into our market are relatively high, right? You’ve got to do it right, because if you want to convince Congress that you need less oversight of the AV industry, you better not be screwing up their travel on a daily basis.

On the depot/workforce concern:

Worried that if we are unable to bring in some of the maintenance and charging aspects of these fleets, that any of the workforce benefits that might accrue associated with this are actually going to go to us, because they’re going to go to Maryland and Virginia, where there is more land.”

“It also has a risk of increasing the zombie miles of empty VMT driving back as they deadhead to get to those facilities.

This shows she’s thinking about empty miles and workforce displacement to the suburbs. She’s not anti-AV here, but she’s clearly concerned about the impacts.

On where they actually are (no deployment framework):

You’ll note that my slides here with the timeline of where we are in legislation and regulation makes absolutely no mention of deployment. We do not have legislation for that at this time.

On the regulatory vs. investment mindset:

I spend a lot of time thinking about what we regulate. I get to spend some time thinking about what we incentivize, and I spend very little time thinking about what we could invest in. I’d like to change that eventually, but first we have to have these base frameworks in place.

She also brings up DC’s objectively weird transit situation:

Lots of places have motorcades. I would argue that we probably have more than most, and they look in all different sizes... it could be everything from the president or the vice president moving on street... but also, we run a lot of unannounced motorcades that are just a couple vehicles moving through, as perhaps a senator is coming up to Congress. There’ll only be two or three black cars running all sorts of red lights.

We are the central city in a tri-state region... on the one hand, we are a state, but at the same time, we’re also at the very middle of a region, and our region involves Maryland and Virginia, and depending on the commute shed that you’re looking at goes as far as West Virginia as well.

DC does not... is not an industrial city at all, right? That’s not why we exist. So we have a real shortage of land that can be used for anything like a depot. We don’t have a lot of big warehouses.

DDOT seems to have expanded the scope well beyond what the law required. The July 2024 Wayback Machine snapshot of the AV Working Group page says DDOT was trying to “procure consultant support to help the agency define a potential regulatory and legislative framework for the deployment of AVs.” That’s already broader than “recommendations” the original law called for. And Allen has added on additional expectations. In his public statements he’s talked about wanting the report to address job displacement, worker transition, and what he calls getting it “right” in a way that clearly extends beyond the safety focus of the original mandate. His December 2025 newsletter explicitly raised “the impact on jobs” as something the report needs to address. The law never called for that.

It’s crazy that after almost four years, we don’t know for sure if DDOT is planning to actually provide a permitting system along with the report. It would be nice if they would just make the general goals for the report clear, because even there we have very little to go on besides the original very short mandate.

Charles Allen

Allen is the single most important person on the legislative side. He’s the chair of the Committee on Transportation and the Environment, so he controls whether any AV bill gets considered. His making the DDOT report a precondition has given the department a total veto over the legislative timeline.

Allen has been careful to say he’s not opposed to AVs. He’s said he doesn’t “think we should be afraid of this technology or throw up obstacles” and that autonomous vehicles will “likely operate in the District” in the near future. But the practical effect of his position is indefinite delay.

His stated reasons are about process and safety. From his December 2025 newsletter:

The idea that we should authorize fully driverless vehicles without DDOT’s analysis in hand is not safe, nor is it even guaranteed to happen if the Mayor and DDOT don’t want to issue permits.

He’s also raised job impacts:

The questions also need a stronger conversation about the impact on jobs — DC residents working as taxi and rideshare drivers today would be replaced, and the money you pay for a ride would no longer go back into the community but rather be sent out of state to corporations largely located in California.

That one sounds especially bad, because there’s no way that Waymo (or buses, or bikes) could be legalized without reducing the number of rideshare jobs at least a bit.

Some other key quotes from Allen on how he’s thinking about this:

Autonomous vehicles hold promise, but DC residents should not be treated like test subjects while companies work through unresolved issues on busy city streets. Given recent nationwide examples of AVs making dangerous mistakes, a careful, data-driven approach is warranted. The questions also need a stronger conversation about the impact on jobs – DC residents working as taxi and rideshare drivers today would be replaced, and the money you pay for a ride would no longer go back into the community but rather be sent out of state to corporations largely located in California.

We’re committed to getting this right – advancing innovation without compromising safety – and to making sure autonomous vehicles are integrated thoughtfully into a transportation system that works for everyone.

- December 2025 newsletter

I don’t think we should be afraid of this technology or throw up obstacles. But my job isn’t to say, ‘How fast can I roll this out?’, but rather, ‘How can we get this right?’ I’d rather get it done right than get it done fast.

- Interview with The 51st

(in response to a claim that AVs are involved in more crashes than human driven cars) This to me is a great example that I don’t necessarily want to be the first into this marketplace. I don’t mind having other cities go first but let’s learn what works and what doesn’t and then figure out what’s best for D.C.

- Interview with FOX 5 DC

Could he change his mind on making the report a condition? Nothing legally prevents him from holding hearings without the report. McDuffie’s bill exists and could be brought to hearing. But Allen has made this a matter of principle, and I’m skeptical that the relatively small number of AV advocates in DC can change his position on it. I also personally just have no way to judge how reasonable his decision is. There’s a legitimate case that the council should have expert analysis before writing complex regulatory legislation. What I can say is that DDOT’s three year delay has turned what might be reasonable caution into an effective block. I do worry a lot about a lot of the quotes he’s shared here on safety and jobs. Both go way beyond a reasonable or fair assessment of Waymo compared to any other vehicles.

Allen has no control over DDOT, but someone else does…

Mayor Bowser

Mayor Bowser also bears responsibility here. DDOT is an executive agency that reports to the mayor. The report is almost four years overdue on her watch. Bowser could direct DDOT to prioritize it. She could also direct DDOT to separately begin designing and issuing driverless testing permits.

Allen has publicly blamed Mayor Bowser for not issuing driverless testing permits. In December 2025, he told Axios “I don’t know what the holdup is.” Bowser’s office responded to a media request on this anonymously. An official who wasn’t authorized to speak to the press told Axios “It’s not us” and characterized DDOT as “being super diligent with their research.” I’m not sure what to infer from this.

The fact that DDOT has been allowed to sit on this for nearly four years without consequence suggests either that Bowser doesn’t consider AVs a priority, or that there’s an active decision to keep this in limbo.

Bowser could unblock the situation from the executive side without needing the council to do anything. Getting DDOT to produce the report would remove the justification Allen is using to hold up legislation.

Bowser has less than a year left of being mayor after 3 terms. A new mayor will take office in January of next year, and the primary that will determine who will be mayor is happening this June. Bowser hasn’t given any sign that she’ll give any priority to AVs in her last year as mayor, so while I wish she’d use her executive authority to push DDOT to prioritize the report more and to separately issue driverless testing permits in the meantime, I don’t expect this to happen. She doesn’t have much incentive to push something new and controversial like this.

Takeaways

• Waymo needs two things from DC’s government to operate: testing permits for Waymos to drive without humans inside, and legalization of commercial AVs.

• The report is the main political bottleneck for both. It blocks legislation because Charles Allen, the transportation committee chair, won’t hold hearings without it. It’s also part of the justification DDOT gives for not moving forward on testing permits, even though the 2020 law established testing permits and the deployment report as two separate obligations, and it’s not even clear the report will address testing permits at all.

• But the report isn’t the only bottleneck. DDOT’s unfinished permit and rulemaking process (a separate track from the report) is the practical bottleneck for driverless testing. The driverless testing permits could also be produced if Mayor Bowser decided to direct DDOT to open the permit pathway that already exists in statute, but it looks like she won’t, and DDOT hasn’t given any indication that they’re separately working on driverless testing permits.

• There are recent signs of progress. DDOT told both Axios and Waymo the report could be done by summer 2026, and they appear to have contracted outside consultants. But DDOT has missed deadlines before and is more than three and a half years late.

The council and mayor debate once the report is published

Once the report is published, this will only clear the way for the DC council to actually make policy on AVs, and we know almost nothing about what each DC councilmember believes. Many other cities in the Northeast have delayed or rejected self-driving cars.

Of the 13 councilmembers, the only one besides Allen who has been publicly vocal about Waymo is Janeese Lewis George, and she’s been skeptical of AVs (more on her below, note that she’s the other main candidate for the next DC mayor). Everyone else seems to not have commented at all. I cannot find any hints of what any of the councilmembers think about Waymo.

One piece of info we do have is who voted for the 2020 AV bill that created the original AV framework and gave DDOT authority over AVs. The bill seems pretty pro AV, it was passed unanimously, and 7 of the 13 councilmembers who passed it are still in office, though some are not running for re-election and will be out of office in January 2027:

Current councilmembers who voted for the 2020 bill:

• Phil Mendelson

• Anita Bonds (not running for re-election)

• Robert White (running for Congress)

• Brianne Nadeau (not running for re-election)

• Brooke Pinto (running for Congress)

• Charles Allen

• Trayon White (currently awaiting a federal bribery trial scheduled for September, creating an unprecedented dynamic where he was unanimously expelled from the DC Council in February 2025, only to win his Ward 8 seat back in a July 2025 special election)

The six councilmembers who weren’t in office in 2020 are:

• Janeese Lewis George

• Christina Henderson

• Doni Crawford

• Matt Frumin

• Zachary Parker

• Wendell Felder

If a councilmember hasn’t thought about AVs, this might be a great opportunity for DC citizens and groups to influence their opinion early before the issue becomes polarized, or before the anti-AV interest groups reach them first. This actually seems really urgent in a way that hasn’t been covered as much, because even when the report is passed, the decision to legalize Waymo will be in the hands of these 13 councilmembers, and the only one who’s spoken about Waymo specifically has been negative. I think we should actually be somewhat pessimistic about how the council will vote. As noted in Understanding AI, Waymo has had a lot of success in Republican-dominated states where the regulatory environment is more favorable, and has hit barriers in more Democratic states. As a baseline, because DC skews very left, and this pattern of delay in more left of center places is becoming more consistent, we should expect a higher chance of bumping into more political opposition here.

One other note is that the two council votes running for Congress (Pinto and White) will be heavily influenced by how organized labor (a vital constituency in a citywide/district-wide congressional primary) views the AV issue.

The mayoral race

I really don’t think people should vote for DC mayor based on how they think about AVs, but it’s also important to understand that the two main candidates for DC mayor seem to be on opposite sides of the AV debate and it’s likely that this election will be extremely consequential for the future of self driving cars in the city.

The mayoral race in DC is basically decided by the Democratic primary on June 16 2026. Because DC is overwhelmingly Democratic, whoever wins that primary will almost certainly be the next mayor, taking office in January.

The two leading candidates in the primary are Janeese Lewis George and Kenyan McDuffie. McDuffie was the most pro-AV member of the DC council, and Lewis George is the most anti-AV.

Here’s a prediction market for who will win:

Kenyan McDuffie

McDuffie is the former councilmember who introduced the Autonomous Vehicles Amendment Act of 2025 (B26-0323), the bill that would create DC’s complete AV operating framework. The bill was introduced on July 11, 2025 and referred to the Committee on Transportation and the Environment on July 14, 2025. It would allow fully autonomous vehicles to operate without a human driver; establish insurance and financial responsibility requirements; require companies to submit law enforcement interaction plans; create an on-demand AV network under existing for-hire transportation law, with human-driver-only provisions carved out; and designate DDOT as the sole agency implementing the act.

McDuffie was regularly described as one of the most “pro-business” councilmembers. If elected mayor, he would likely direct DDOT to finish the report on a hard deadline, possibly issue driverless testing permits independently, and use his relationship with Allen and other councilmembers to push for hearings.

Surprisingly, McDuffie hasn’t done or said anything else related to AVs that I can find.

Janeese Lewis George

I want to flag that I’m going to be pretty critical of Lewis George’s coalition’s attitude toward AVs here, but I’m otherwise unsure of who I’m going to vote for and friends I trust have wildly different takes on which candidate is preferable in general, though basically everyone I know agrees that we’d like Lewis George to change her mind on AVs. Among other things Lewis George got the endorsements of prominent housing groups in DC who I tend to agree with.

Lewis George is the councilmember who has been most publicly skeptical of AVs. She said in a podcast interview in February “I don’t think our city is ready for Waymo at this moment.” She’s framed her concern as primarily about safety, but she also spoke about worker displacement, saying “we’re gonna create a balanced approach” because “I want to make sure we are not displacing workers.”

Lewis George describes herself as a democratic socialist and has built her career around labor advocacy. In December she got early endorsements from five unions with significant local organizing presence: SEIU 32BJ (which represents property service workers including janitors, security officers, and airport workers), ATU Local 689 (transit workers), UFCW Local 400 (grocery, retail, and healthcare workers), and Unite Here Locals 23 and 25 (hospitality and food service workers). At that endorsement rally, she pledged that as mayor she would “take on companies” engaged in union-busting and protect workers from displacement fueled by artificial intelligence. The Working Families Party and Washington Teachers’ Union have also endorsed her.

Axios reported that “a prominent local labor union” endorsed Lewis George in part because of concerns about job losses from autonomous vehicles specifically. The Washington Informer identified this union as SEIU 32BJ (property service workers), whose executive vice president Jaime Contreras said they endorsed Lewis George because "She's anti-Waymo, which is something that we oppose coming to the city because it's going to displace working-class people who do Uber or Lyft or DoorDash…for extra income.” This mirrors the pattern in other cities where organized labor has emerged as the primary opposition force to Waymo’s expansion.

Lewis George hasn’t called for banning Waymo or said autonomous vehicles should never operate in DC. Her stated position is that the city isn’t ready “at this moment” and that she wants to ensure workers aren’t displaced. This sounds reasonable, but is actually quite extreme as a reaction to new vehicles. Any rule that says we can’t introduce a new transit technology if it threatens to cause rideshare drivers to lose their jobs is an abysmal way to think about transit policy. Among other things, if buses didn’t exist, this same rule would ban buses. This same rule would have also banned rideshare apps in the first place, because they mostly destroyed the (better regulated, better paid, easier to organize) taxi industry in DC1. I have more on that argument here.

Lewis George’s position on Waymo closely matches the positions of leaders in cities that have stalled or effectively banned it:

• In Boston, the city council’s proposed ordinance uses identical framing: study the impact on jobs first, require a human safety operator in every vehicle, and don’t allow commercial operations until an advisory board gives the green light. That advisory board would be required to include representatives from the App Drivers Union (a rare example of unionized rideshare drivers), the Greater Boston Labor Council, the Teamsters, and the UFCW. Eight city councilors collaborated on the ordinance. The Teamsters organized a coalition called “Labor United Against Waymo” which was the first mass-mobilization effort outside California to push back against AV expansion. Waymo’s spokesperson called the proposal the “first major city in the world to ban fully autonomous vehicles based entirely on vibes.”

• In New York, Governor Hochul withdrew her robotaxi legalization proposal in February after failing to secure legislative support. Her spokesperson said “Based on conversations with stakeholders, including in the legislature, it was clear that the support was not there to advance this proposal” which in practice meant that organized labor made the political cost too high in an election year. The proposal would have allowed limited robotaxi deployment upstate while leaving New York City decisions to the mayor and city council. It died anyway. Waymo has testing permits for a handful of vehicles in Manhattan, but can’t charge passengers or operate commercially without new legislation that now doesn’t exist.

• In Minnesota, SEIU chapters started organizing against AV legislation before Waymo even announced plans to expand to the Twin Cities. By the time Waymo began testing, there was already organized opposition pushing for legislation that would require human drivers in every vehicle. Multiple Minneapolis city councilmembers have signaled interest in passing a citywide ordinance with the same requirement.

• In Washington state, law allows testing but requires a human safety operator in all vehicles and doesn't offer a clear path to commercialization. Seattle has its own additional testing permit program on top of the state requirements.

The same pattern shows up again and again where politicians express openness to AVs in principle, insist on studying job impacts first, and structure the study process so that it can never actually conclude that it’s okay to proceed. The “study jobs first” framing often functions as a ban.

Lewis George hasn’t gone as far as Boston’s councilors, who proposed requiring a human in every vehicle. But her framing seems to track the same playbook, her coalition includes the same types of unions driving opposition in those other cities, and the political incentives point in the same direction. The Teamsters’ national position is a permanent human-operator requirement for all autonomous vehicles. ATU Local 689, which endorsed Lewis George, has lobbied against AVs alongside its national chapter.

That said, Lewis George does have room to eventually support AV deployment. She could condition it on workforce transition programs, labor protections, or union consultation requirements. There’s a version of AV legalization that includes strong worker provisions and that she could plausibly support without reversing her position. The question is whether she’d actually pursue that as mayor, or whether the political incentives from her labor coalition would push her toward indefinite delay.

This doesn’t mean AVs would never happen under Lewis George. She hasn’t called for a permanent ban, and external pressure would keep building: Waymo would be operating in 15+ other cities, Baltimore might have the service 40 miles away, federal legislation could change the landscape. But nothing in her current positioning suggests she’d prioritize this, and the most likely outcome is that legalization gets pushed back by multiple years.

The mayoral race makes the DDOT timeline much more important, and vice versa

The mayoral race matters for AVs, but how much it matters depends almost entirely on whether the DDOT report comes out before the new mayor takes office in January 2027.

If the report comes out before January 2027, the mayoral race becomes potentially less decisive. Allen’s stated precondition would be removed. The McDuffie bill already exists. Allen could hold hearings in the fall. The council could potentially vote before the end of the year. If legislation passes before January, the new mayor inherits a legal framework, and unwinding legislation the council has already passed is much harder and more politically costly than just preventing it from ever being introduced.

Bowser could also use the report as cover to issue driverless testing permits in her final months. If DDOT hands her a finished product, the political calculus changes. She could claim credit for moving the city forward on her way out.

If the report still isn’t finished by January 2027, the mayoral election becomes the single most important variable in whether Waymo is legalized in DC anytime soon. The next mayor controls DDOT, and the report’s absence is what keeps everything frozen. Under Lewis George, the stall could extend for years, especially if she adds requirements to look into effects on jobs and halts legalization until there are guarantees that rideshare drivers won’t lose work.

Under McDuffie, things would likely move faster, though he’d need to navigate the council’s labor concerns.

This is why pushing for the DDOT report to come out before the election is the single most important thing AV advocates can do. Every month of delay past this summer makes the mayoral outcome matter more.

The federal government

Congress has direct authority over DC. While the local political dynamics I’ve described above are the most important near-term variables, federal action could eventually change the landscape, though right now there are no especially promising bills. Two bills that would in part pre-empt DC’s decision making on AVs are the SELF DRIVE Act of 2026 and the Autonomous Vehicle Acceleration Act of 2025 (S.1798), but both look very unlikely to become law. The SELF DRIVE Act is being talked about as maybe possibly being folded into the Surface Transportation Reauthorization, which would make it more likely to pass, but this is all pretty speculative. Technically the federal government could eventually pre-empt DC law if they passively disallow or actively ban Waymos, though I wouldn’t count on it anytime soon.

How to legalize Waymo in DC

The most promising levers to pull

I don’t know a lot about the specifics of influencing city government, so take these with a grain of salt. I’ll update them as I get feedback. The three main things to do right now seem to be:

• Apply public pressure to DDOT to finish the report on time, and then to create the separate necessary testing permit framework. This could be through public writing or writing to representatives.

• Make a plan for how to sway the councilmembers who haven’t expressed any opinions on AVs to support a legal framework for AVs in the city once the DDOT report is finished. It’s likely that like other cities that are pushing back against Waymo, interest groups will be eager to reach undecided councilmembers as well. They’re probably already trying to.

• Push for “shall issue” permits in any AV legislation. This might be the most important detail to get right, and I haven’t seen it discussed publicly. The 2020 law gives DDOT discretionary authority over testing permits. DDOT decides whether to make the application available, how to evaluate it, and when to approve it. There’s no requirement to actually issue a permit once criteria are met. This is exactly how we got here: the law says DDOT can create a permit program, DDOT chose not to, and nothing can force them.

  If the council passes AV legislation and keeps this discretionary structure, the same thing happens again at the deployment stage. A hostile administration sets up a permitting process that technically exists but moves at whatever pace it wants. Applications sit in review. Additional information gets requested in endless rounds. The framework exists on paper while no permits are issued. DDOT’s own AV program manager said in a 2025 MIT panel that “the industry is changing a lot faster than our rulemaking structure is allowing us to keep pace” and that’s without a mayor who’s actively trying to slow things down.

  The solution would be that once an applicant meets clearly defined conditions (safety benchmarks, insurance, data reporting, law enforcement plan), DDOT must issue the permit within a defined timeframe. No discretion on qualified applicants. No sitting on applications.

  Under McDuffie this probably doesn’t matter much. Under Lewis George it’s the difference between a framework that works and one that becomes another tool for delay. Someone with more knowledge of DC government than me pointed out that this might be the most important thing the council could do besides passing the legislation itself. I think they’re right. If the legislation gives the executive branch unlimited discretion over whether to actually issue permits, we end up right back where we are: a legal framework that exists in theory and doesn’t work in practice.

These seem less promising:

• Convincing Charles Allen to change his mind on using DDOT as a prerequisite for any policy. Allen seems to have made up his mind on this, I’m skeptical that he can be swayed with the admittedly limited number of AV advocates in DC, and I also personally just have no way to judge how reasonable his decision is. It does make sense to me that councilmembers ought to rely on government agency expertise. A lot of what he’s shared on DC looks silly and overly critical of the cars compared to most other vehicles though, and the standards he’s using to judge them (jobs and safety) would imply every other type of vehicle in the city shouldn’t be legalized. On that maybe we can make some progress.

• Relying on Mayor Bowser to take action. She doesn’t seem interested in prioritizing AVs and I’m not sure how much this could change.

Can we change Lewis George’s mind?

A strategy I’m unsure of is convincing Lewis George to change her mind on her current stance on AVs. She says she wants to ensure rideshare workers aren’t displaced, but this same rule would also ban buses or bikes. Her union endorsement coming in part because of her stance on AVs makes it seem difficult to change her mind. That said, Lewis George describes herself as a democratic socialist, and there are arguments for AVs that map onto things democratic socialists care about:

• Fewer traffic deaths. DC sees about 40 traffic fatalities a year, disproportionately in lower-income wards with fewer protected pedestrian crossings.

• Every Waymo is electric. DC has climate goals and environmental justice commitments. Waymo’s fleet is entirely electric, while a significant share of Uber and Lyft drivers in DC drive gas-powered cars. Replacing gas-powered ride-hail trips with electric AV trips is a direct emissions reduction, concentrated in the neighborhoods with the worst air quality. For someone who cares about environmental justice, this should matter. It won’t be decisive on its own, but it makes the “let’s wait” position slightly more costly in environmental terms every month it continues.

• Ride-share jobs aren’t union jobs and probably never will be. This is the argument I find most interesting, though I’m least sure it would land politically. The jobs Lewis George is protecting (Uber and Lyft driving) are difficult-to-unionize gig work. They’re not the kind of jobs that democratic socialists typically celebrate. To my knowledge, DC does not have any rideshare unions. These are rare in general but do appear in cities where labor is especially powerful, like Boston and Seattle. It’s likely that the unions pushing against Waymo are actually worried about future automation of buses, trains, or trucks, and see automated rideshares as the beginning of a wave of automation that will eventually come for them too.

• Constituencies who otherwise have trouble getting around the city might benefit from AVs. Elderly people among others.

I want to be honest that I don’t know if any of these arguments would actually move Lewis George. Politicians rarely reverse positions that are tied to endorsement relationships, especially during a campaign. The more realistic hope might not be changing her mind before the election, but rather creating enough cross-pressure from interest groups, environmental advocates, and safety organizations that, if she wins, she feels politically constrained from actively blocking AVs indefinitely.

The reason this matters is that if Lewis George’s position is immovable, then whether AVs are legalized in DC hinges almost entirely on the June primary. Any progress on softening her stance would reduce the stakes of the election and make the path to legalization less dependent on a single vote.

Lessons from other cities

Every city that has Waymo got it because at least one of three conditions was met:

• A state-level framework that preempted local objections (California, Texas, Arizona)

• An executive who personally championed the technology (Phoenix, Atlanta, Denver)

• A regulatory process that moved on a defined timeline (California’s phased CPUC system)

Unfortunately, DC currently has none of these.

Here are some case studies:

California

San Francisco’s city government was deeply skeptical of Waymo. Its transportation agencies formally protested expansion, and the city actually sued the California Public Utilities Commission to try to block it. The CPUC approved Waymo’s commercial deployment in August 2023 over San Francisco’s explicit objections, and a state appeals court upheld the decision.

Public opinion in San Francisco shifted dramatically once the service was actually available. In mid-2023, net favorability for robotaxis was negative 7%. By mid-2025, it had swung to positive 38%, with 67% of residents supporting continued operations. People tend to like AVs once they can actually try them.

Texas and Arizona

Both states created permissive frameworks before Waymo arrived. Arizona’s governor issued an executive order welcoming AV testing in 2015. Texas passed legislation in 2017. When Waymo was ready to launch, the legal framework was already there. DC’s 2020 law was supposed to be that proactive step. DDOT’s failure to follow through is why DC is now years behind.

Boston: The closest parallel to DC

Boston is the most instructive comparison. The Teamsters organized “Labor United Against Waymo,” the first mass-mobilization effort outside California to push back against AV expansion. Eight city councilors collaborated on an ordinance that would require every AV to have a human safety operator (effectively banning the whole thing) and commission a study before any commercial operations could begin. The advisory board overseeing the study would be required to include reps from the App Drivers Union, Greater Boston Labor Council, Teamsters, and UFCW.

The Boston situation is still unresolved. Competing state-level bills are being debated. One allows driverless vehicles, the other requires human operators in all vehicles. Waymo has returned to Boston for further testing but faces serious organized opposition.

This Boston city council meeting drove me completely crazy to listen to, and might be a hint of the bad dynamics at play in coming debates about AVs in DC.

Minnesota

In Minnesota, SEIU chapters started organizing against AV legislation before Waymo even announced expansion plans. By the time testing started, there was already an organized opposition pushing for human-operator requirements. This is almost certainly already happening in DC.

Maryland

Governor Wes Moore is publicly enthusiastic about Waymo. The Maryland legislature is actively considering bills to allow fully autonomous vehicles, and hearings have gotten surprisingly little pushback from lawmakers. Waymo already has cars in Baltimore being driven by professional drivers to map the city. If Maryland passes its legislation, DC residents will be able to see the technology operating 40 miles away while they can’t use it at home. Maybe this will have some impact?

Atlanta and Austin

In both cities, Waymo launched partly through a partnership with Uber, making rides available through the existing Uber app. This partially neutralized the “Waymo vs. rideshare drivers” framing by positioning Waymo as an option within the existing ecosystem rather than a replacement for it. Whether this model could help politically in DC is unclear, but the narrative doesn’t have to be adversarial.

The national labor strategy

The Teamsters have called nationally for a regulatory standard requiring a human operator in all autonomous vehicles, with that operator subject to commercial driver’s license requirements. Teamsters president Sean O’Brien testified before Congress that “allowing the unfettered and unregulated operation of autonomous vehicles, ultimately seeking to replace human drivers with robots, is unequivocally a threat to safety on our roadways and the existence of good jobs in the trucking industry.” The labor impacts framing is part of a coordinated national strategy.

Federal momentum is actually moving in the opposite direction from the labor strategy. The SELF DRIVE Act passed the House Energy and Commerce subcommittee on a narrow 12-11 vote and could be folded into this year's Surface Transportation Reauthorization. Congress seems more receptive to AVs than blue-city local governments are. There’s no serious national ban proposal on the table. But labor opposition is likely to grow.

DC matters in the federal picture for a specific under-discussed reason: it’s where members of Congress, their staffers, federal regulators, and lobbyists live and commute. If Waymo operates in DC and these people experience it working well in their daily lives, it might normalize the technology among the people who write federal policy. If it’s anything like the other cities where Waymo’s operating, we can expect the public perception of AVs among policymakers to meaningfully improve.

Subscribe now

This talking point is crazy. It’s so obviously wrong that it’s disorienting to me whenever I see grown adults saying it. This is one of the pieces of common wisdom that’s congealed about AI that collapses when you poke at it for even a moment.

Knowledge is justified true belief.1 Can AI give me a justified true belief that no one else has had before? Yes, obviously. Google searches can also generate new justified true beliefs. It’s not actually difficult at all to generate new knowledge. It doesn’t require magic.

Subscribe now

Here’s a very simple example: I asked Claude “Please do a BOTEC of how many pieces of avocado toast in restaurants in DC equal the cost of renovating DC’s Union Station”

This has given me a new justified true belief. It’s justified because it provides sources, and I can clearly see that there are good reasons to believe each of the premises that lead to the conclusion that the Union Station renovation costs roughly half a billion avocado toasts. It appears to be true. And it’s entered my mind as a belief I hold. And it’s new. I don’t think anyone’s held this specific justified true belief before. AI can create new knowledge. Why would anyone say otherwise?

This could have all been done with 2 Google searches: the cost of the Union Station renovation, and the cost of avocado toast. Doing a few Google searches can also generate new knowledge. It’s very very very easy to generate new knowledge! Why would AI models as they exist not also be able to do this?

I think that when people say this, they don’t actually mean new knowledge, they mean new concepts. AI has been trained on basically the full corpus of human text, rewarded for adhering to our conceptual universe, and punished for straying from it. It’s hard for me to imagine a deep learning model as they exist independently inventing the concept of a negative number if its training data had not ever included any implication that negative numbers exist.

So I probably agree with LLM skeptics that there’s a good chance they don’t scale to AGI, because the way they’re trained might mean that they’re fundamentally not able to generate radically new concepts for thinking about the world, which seems like a prerequisite for full human intelligence. They can merely approximate almost all currently existing human concepts that can be expressed in text, and combine wildly disparate information using those concepts to help us generate huge amounts of new knowledge. Still seems useful to me.

Subscribe now

Subscribe now

Louis Anslow who runs the Pessimists Archive coined a useful simple extension of this idea: the lump of cognition fallacy: the idea that there is a fixed amount of thinking to do. I see this come up a lot in AI discourse. Like the lump of labor fallacy, the reason this is wrong is simple: thinking often leads to more things to think about.

I was recently sent a funny article in the Guardian about how the author isn’t dating anyone who uses chatbots in part because they’re bad for the environment. Articles like this generate a few easy clicks for my posts. But I was struck by the author’s broader complaint, shared by many people she interviewed, that AI somehow causes people to think less throughout the day. That it’s literally always intellectually lazy to use. Take this quote:

Pereira thinks that using ChatGPT “shows such a laziness”.

“It’s like you can’t think for yourself, and you have to rely on an app for that,”

The article lists a few things that I agree are bad to use ChatGPT for, like writing messages on dating apps. But the author extends this to strongly imply that using ChatGPT at all is causing people to think less, because any cognition the chatbot performs leaves the user with fewer thoughts to think. I would brush this off as silly rage bait if I weren’t also so regularly bumping into the same idea elsewhere. I’ve written about this a bit before, but really want to get across just how strange and obviously wrong this is as a model of how human thought works. I also want to popularize Louis’s name for it. I find myself muttering “the lump of cognition fallacy” pretty regularly now. I want to get across a few intuitions about why there isn’t a limited amount of cognition to do, and why outsourcing cognition is often just a way of allowing yourself to think more deeply and meaningfully rather than avoiding thought.

Is it lazy to watch a movie instead of making up a story in your head?

Choosing to watch a movie is a massive outsourcing of thought. You could just close your eyes and imagine the whole thing in your head. There are so, so many mental tasks involved that you’re choosing to have someone else do for you:

• Making up a story

• Writing the music

• Choosing the lighting, set, decorations, costumes, etc.

Have you ever seen someone watching a movie and thought “they’re so lazy for not just imagining a story?” If not, why?

The reason is that there is not a limited amount of thinking to do about the story. The fact that other people have put in so much mental effort into so many parts of it opens up way more potential ways to think about it compared just making up a movie in your head, for the same reason a large complex economy has way more job opportunities than a simple small economy, even though in the large economy more work has “already been done.”

I remember watching Lord of the Rings as a kid and being totally overwhelmed. I didn’t think about anything else for days. The fact that so many adults had already done so much complex thought about the movie, thought that wasn’t available to me as a kid, just meant that my own thoughts about it exploded in a ton of directions they wouldn’t have if I were just reacting to something I made up in my head. Just like we benefit from specialization in labor, I was benefiting from the cognitive specialization of people who had spent decades thinking about story, images, music, and sets, and this left me with way more things to think about, in the same way the wild level of economic specialization that goes into making a computer just leaves you with way more you can do with it after. Similarly, buying a pencil is a way of outsourcing physical labor to thousands of other people. Just like outsourcing labor to other people is mainly a way to allow yourself to do more in total (buying a good laptop vs. trying to build one yourself), outsourcing cognition to a movie leaves you with way more high-level thoughts to think.

The extended mind

This leads into one of my favorite ideas from philosophy of mind, the extended mind thesis: much of our cognition isn’t limited to our skull and brain, it also happens in our physical environment, so a lot of what we define as our minds could also be said to exist in the physical objects around us.

One simple example is how we use our phones to store information. Storing our friends’ phone numbers is similar to storing them in our memory. In both cases, they mostly stay out of our conscious minds. When we want to retrieve them, they are easily available. Over time, dialing a friend’s phone number might become so second nature that you don’t actively think about it. In the same way, having the number stored might mean you never actually have to look at it, you just select the friend to call. In both cases, you’re causing something like cognition to happen in order to achieve some goal, most of it not happening in your conscious experience. It’s behaving as if it’s a part of your broader mind. It seems kind of arbitrary whether it’s happening in the neurons in your brain or in the circuits in your phone. It’s true that you could lose your phone and therefore lose the stored knowledge, but you could also have a part of your brain cut out. Our thinking is often supported by hugely complex background processes both in our skulls and outside of them. The extended mind seems like a useful analogy to help us see how to get the most out of our thought. Offloading our thought onto the environment is in some sense a method of radically expanding and freeing up our minds.

The built environment

To get more extreme, most of our physical environments have been designed specifically to minimize the amount of thinking we have to do to achieve our daily goals. The reason many parts of society are standardized is to reduce the cognitive load of navigating them. Grocery stores are all laid out in similar ways, keyboards are arranged in the same order, street addresses are systematized clearly, all to prevent us from having to expend precious conscious thought on navigating them. When I approach a door, I have no conscious thoughts at all about how to open it. My active thinking is occupied by other stuff, my arm just subconsciously reaches out to the correct location. In these moments, I have “outsourced” my thinking to the door design, because there was a possible world where I had to actively think about opening the door, and the reason I don’t is how the door was crafted.

Try just walking around your daily life and observing how much of the built environment exists to minimize the amount you need to consciously think about getting what you want. Try to imagine how much additional thinking you would need to do if things were designed differently. You’ll quickly notice that huge swaths of our lives are designed to quickly become intuitive to us, so that we don’t constantly need to occupy our thoughts with navigating our built environments or figuring out simple tasks like making payments or traveling or eating. The fact that I have outsourced so many possible thoughts to my built environment liberates me to think about higher level stuff, the things I actually find deeply valuable about the world. I can spend more time thinking about my friends or my job or interesting ideas or media I like or just pausing to take in the beauty of a tree or a building. I don’t need to constantly plan out how to move through the world, because those decisions have often been “outsourced” to civilization.

Other people

Thought is incredibly social. We learn through imitation. Thought uses shared language and symbols. Culture transmits old tacit knowledge. Every thought I’ve had has just built off the complex thoughts of other people. In some sense I’ve offloaded the learning required to be part of an advanced civilization to all prior humanity. This is all extremely positive-sum. There isn’t less to think about now that humanity has thought through the last 10,000 years of transition and left us with the results at the end.

Institutions

More broadly and abstractly, much of the “reasoning” happening in society in any one time might actually be happening via the mechanisms of market or government institutions or culture, evolving over time, maybe to achieve some higher way of being. This is kind of what Hegel was talking about.

Civilization as offloading thought

This is all so obvious it seems silly to write, but it actually doesn’t make sense under the lump of cognition fallacy. Our ancestors had to expend way more cognitive power just navigating their day to day lives. The modern world has more or less completely removed that need compared to most of human history. With so little of that conscious cognition happening, people who believe in the lump of cognition should predict that people just think way less often now than they did before. But that’s clearly not true. With our minds freed from huge amounts of minor inconveniences, we can spend our days thinking about more interesting stuff. The philosopher Whitehead described this process well when he said:

Civilization advances by extending the number of important operations which we can perform without thinking of them.

I think people who worry about how chatbots literally always involve outsourcing some mental task might not be noticing the gigantic mountain of mental tasks we have already outsourced to civilization, and how this has only liberated us to think more deeply and meaningfully more often. The lump of cognition fallacy causes people to see this backwards: instead of cognition leading to more and more thought, they see it as draining a finite pool, and nothing will be left to think about once it’s drained.

When is it good or bad to outsource thinking to chatbots?

So there are plenty of examples of where it’s good to offload our cognition, because that only frees us up to live more meaningfully and think in more complex ways about what actually matters to us. But there are also clearly cases where it’s very bad to offload our cognition. Things like:

• Homework.

• Messages on dating apps.

• Summarizing a valuable complex book instead of reading it (assuming you had the time and energy to read it and would have benefited from it).

• Personal connection and close conversation.

What’s the difference between these and all the good ways we have outsourced our thinking? When is it bad to outsource?

It seems pretty obvious to me that there are actually a few key obvious places where outsourcing your thinking is bad. They all overlap with places where outsourcing your labor is bad. It’s bad to outsource your cognition when it:

• Builds complex tacit knowledge you’ll need for navigating the world in the future. Homework is valuable because it trains your brain to see more and more complex patterns in the field you’re trying to learn about. Marinating in cognitive work in a field can give you subtle, hard-to-communicate knowledge about it, or at least gives you knowledge you can easily recall in the future instead of having to look it up. It’s bad to outsource cognition on homework for the same reason it’s bad to hire someone to take your place as a student in a class.

• Is an expression of care and presence for someone else. If a friend or partner asks for your help with a problem, they often want to feel your mental presence just as much if not more than they want a technical solution to the problem itself. Paying someone else to make something nice for your partner feels less special.

• Is a valuable experience on its own. I wouldn’t want to outsource the thoughts I have after a movie, the simple appreciation of a nice day, or the thoughts in a night with friends. Somewhat outsourcing these can be really valuable (hearing what someone else thinks about a movie might set off my thinking in wildly new directions), but I still want the experience of thinking about these a lot, because the experience of thinking about them is itself valuable and nice.

• Is deceptive to fake. If someone’s messaging you on a dating app, they want to know what you’re actually like.

• Is focused on a problem that is deathly important to get right, and where you don’t totally trust who you’re outsourcing it to. I don’t hand off very high-stakes decisions to chatbots.

The places where it’s good to offload your cognition to chatbots, where I use them over and over again, are the places where it’s good to offload cognition in any other circumstance, where we rely on the built-up knowledge of other people and civilization, where I won’t gain anything from putting in cognitive effort, and where there are massive positive-sum spillover effects of other people being involved in the thought process.

I was recently around someone complaining about how people aren’t thinking for themselves anymore because of chatbots. They had mentioned that multiple people they knew were using them to find recipes and make grocery lists. Most of my best cooking experiences have been following recipes I find online. My willingness to offload the mental effort of putting the recipe together means I get to do much more ambitious things in my cooking. I don’t think this person would have ever been upset if I had mentioned I Googled a recipe, but because a chatbot was involved, the lump of cognition fallacy was operating, and they had the sense that having a recipe given to you took away a finite opportunity to think for yourself.

Among other things, I use chatbots as research assistants. They do the tedious, repetitive work of assembling sources on a relevant topic, I read and check the sources themselves. I get the benefit of reading about and thinking through the key ideas, they do the drudgery of long boring internet searches and digging through academic literature for relevant information. For the same reason human research assistants don’t turn humans stupid, chatbots aren’t turning me stupid either, even though both involve massive offloading of cognitive effort. They free me up to do the type of thinking that actually helps me learn a lot about the world. In all the other ways I use them, they imitate a good conversation with a subject-matter expert, or a simple assistant to arrange text in a way I want. The lump of cognition fallacy implies that anyone using a research assistant is missing out on some of the thought available for their project. Using the research assistant must make them stupid and lazy. But obviously this isn’t true. Research assistants leave people to think deeper and more thoroughly about the problems they’re tackling. The incredibly positive-sum nature of thought is obvious when the research assistant is a human. But when it’s a chatbot, people suddenly sense some zero-sum way in which, if the chatbot is thinking, they’re not.

When you’re in conversations about chatbots “replacing our thinking” see if the person is legitimately concerned about the chatbot removing some important experience, or if they’re just operating under the lump of cognition fallacy. Articles about this abound with examples. Take this from the same Guardian article:

Noijeen stopped using AI to code, and uses it very sparingly in his personal life. He will make fun of friends who use it too much. He recently met up with an old friend who lives a three-hour train ride away. They decided to meet in the middle. The friend said he would use ChatGPT to find the right spot, but Noijeen just looked at a map. “There’s a city exactly between us,” he said. “Why do you need to ask ChatGPT for that?”

If you’ve never worried about Google Maps “replacing thinking” but you find yourself reacting negatively to a chatbot offering directions, or you regularly check restaurant reviews before going but find yourself repulsed at asking a chatbot for recommendations, consider that you’re falling victim to this weird selective application of the lump of cognition fallacy that seems to exclusively but consistently appear when chatbots are involved. It depresses me that so many people seem to want to swamp their day-to-day lives in the meaningless cognitive minutia of long fruitless internet searches or working solo without any input from outside, because they fear that if they offload the minutia, they won’t have anything left to think about. There’s a whole world out there.

Subscribe now

It’s been shared in a lot of recent conversations about AI, from AI psychosis to autonomous vehicles to regular bad outputs. Simon Wilson dug into the source here, looks like it’s from an old presentation at IBM.

Subscribe now

I think this is a pretty silly sentiment and is basically incorrect. Consider the following statements:

• “Bill’s been moving a little slow recently compared to slightly younger people. I think the best move is to kill him and throw his body away and replace him with someone who can move about 0.5 seconds faster on each task.”

• “Martha seemed to consistently say incorrect things when we ask her about a topic. I performed neurosurgery on her and discovered the problem in her brain, I rewired it and she’s behaving fine now.”

• “I just learned that Andy ran over a squirrel. This is really terrible and I’m re-evaluating everything I think about how safe he is as a driver. Luckily, we keep cameras all around his car 24/7 so we can figure out what happened. I’ll perform neurosurgery on him to make sure this never happens again, and will keep him off the road in the meantime.”

• “We have this new service where every day 20 million people talk to 20 million representatives of our company. Unfortunately, we recently found out that a few of our employees are crazy and gave the people they were connected to really bad advice, hyping them up when they asked to be hyped up. We’re going to kill all 20 million employees and replace them all with better employees to get that rate down to zero.”

It seems pretty obvious that we actually:

• Hold computers to way way higher standards of accountability than real people.

• Are usually much more able to monitor and detect their bad behavior.

• Have a much easier time being much more invasive in how we deal with their bad behavior.

I think when people say “A computer can never be held accountable” what they mean is “A computer can never be socially punished for a bad decision.” That’s true, but social punishment is just a means to a goal: better behavior or harm prevention. It’s also a pretty clumsy, often useless tool. Many people are socially punished and still behave terribly. If we had the ability to perform neurosurgery on everyone behaving badly to permanently change their behavior, I think this would be seen as:

1. Much more effective than regular social punishment.

2. Deeply evil and totalitarian.

And yet we do the same to computers regularly. Why would we want to “hold computers accountable” when that’s way, way less effective than what we can actually do to them?

I’ve recently seen this image shared a lot in the context of Waymo running over a cat in San Francisco. It seems like human drivers probably run over 75-150 times as many animals when they drive the same distance as Waymo has. Many of these are hit-and-runs. The people whose animals get killed often can’t hold the driver accountable, because the driver speeds away before people can see the license plate. In comparison, it is very very visible when a car is an autonomous vehicle, and the brand is painted all over it. It’s much easier to identify and complain to a massive company. It’s likely that even minor negative media attention will cost them a lot of money, so they have a strong incentive to change their behavior. The cars can be programmed to better detect their surroundings before being let on the road. Meanwhile, human drivers who kill people with their cars are often eventually allowed to continue to drive. If you kill someone while driving drunk in Florida, as long as you don’t have a prior DUI offense, you can eventually drive again! We’re already way, way more lenient with human drivers than we are with AI drivers.

Another funny comparison is AI psychosis. Many, many people have gone crazy using the internet. QAnon, antivax, and getting memed into doing apologia for North Korea are some surprisingly common examples. But that doesn’t stick out to us, because the people are going crazy interacting with content created by other crazy people. These other crazy people get to stay anonymous and never get properly socially punished. But OpenAI can receive huge amounts of condemnation whenever any additional person goes crazy using ChatGPT. Problems with computer applications run by big corporations are much easier to identify, the corporation is easier to pressure to change its behavior, and the applications themselves are easier to reprogram. If only it were as easy to catch all the crazy people online and turn them normal.

There are great reasons to worry about computers having more control in society. Cybersecurity and AI risk among others. But the idea that they’re somehow less “accountable” than people isn’t one of them. Computers shouldn’t be used to dodge accountability, but it’s very hard to do that in general. Unlike humans, it’s fine to keep computers in what’s basically a totalitarian surveillance state and completely reprogram them at the slightest sign of misbehavior. Computers are held to much higher standards of accountability than humans. Let’s keep it that way, but also acknowledge that citizens of totalitarian surveillance states can make better drivers.

Subscribe now

I’m also sympathetic to people who worry that chatbots as they exist are massively overblown, that we’re in a pretty intense hype cycle, that today’s AI models aren’t reliably producing accurate or novel enough results to justify the money and energy spending on them, that scaling is now producing such diminishing returns that something is about to blow up. I still have plenty of experiences with ChatGPT, Claude, and Gemini where they’ll give me results I know to be straightforwardly wrong or made up.

But there’s a third completely different reaction, which I think of as the chatbot moral panic view, that combines three contradictory beliefs:

1. Chatbots are phenomenally stupid, useless, and incapable.

2. Chatbots cannot provide anything of value by definition.

3. Chatbots are demonic. There is something ominous and evil about using them, beyond any measurable clear harm. The person might list a lot of specific harms chatbots cause, but it’s clear that even if all of the problems were solved, the person would still have some deeper moral revulsion. Their concern about specific problems is post hoc and disproportionate to the problems themselves, and goes way beyond their concern about other online tools causing similar problems.

3 contradicts 1 and 2 because it seems hard for something ineffectual to also be deeply evil. 1 contradicts 2 because, if something by definition cannot produce anything of value, it doesn’t make sense to complain that it’s very far down on some axis of quality. It seems like complaining that a rock can barely sing at all.

Yet I keep bumping into people who seem to hold all three beliefs at once.

This is anthropologically interesting. Chatbots feel to me kind of like Wolfram Alpha used to when I was in college, but it’s very hard to imagine anyone having such a strong moral reaction to Wolfram Alpha, or Wikipedia, or YouTube, or any other website or online app that doesn’t clearly involve some direct harm. There’s something really getting to people about the nature of chatbots.

I’ve developed an armchair psychology theory for why this is happening. My confidence in this isn’t high, but it’s been interesting enough that I find myself coming back to it when I’m having conversations about chatbots, so I figured I’d submit it for review here. This is my most hand-wavy post.

Subscribe now

Chatbots remove our ability to associate information with specific high and low-status people

Years ago I lost a friend to a cult. There were clear indicators that he was vulnerable to cult dynamics I should have picked up on ahead of time, including that he was:

1. Very hyper-sensitive to his personal sense of status in any given conversation or situation

2. Not receiving much status in his personal life

3. Lurching toward more and more desperate means of getting it

4. Easily compelled by big flashy contrarian ideas, even if they very easily fell apart when poked at

and this was all exasperated by some underlying mental health stuff.

I remember the first time I had noticed something was deeply wrong. We were walking around Boston and I had made a joke about how I needed to swing by a bookstore to pick up the latest solarpunk manifesto. He suddenly exploded into a 20 minute nonstop monologue about how solarpunk was “fascist” (you can read about it here and judge for yourself) because of a pretty incomprehensible pile of reasons he was implying he had some special deep understanding of that was inaccessible to me. I was too disoriented and unnerved by this to think much about what was actually happening here: my friend was positioning himself as a special arbiter of truth and ethics as a way of feeling like he had conversational power in the moment. He wasn’t making an argument for something being false or unethical. If he were, he would have built some kind of conversational bridge where I could join him in the debate. Instead, he was specifically putting up incoherent conversational walls to imply he had accessed some special Platonic realm of truth I couldn’t possibly access myself. His brain was short-circuiting out of a desire to feel important (a pretty fundamental evolutionary drive) and mashing any buttons it could, like a desperately hungry animal clawing for food.

I realized years later that my friend was actually just doing an extreme version of bad behavior we all indulge in at different levels: setting the boundaries of which people and what ideas are considered high and low-status exclusively to flaunt our social power and importance.

One way this often comes up is in conversations at parties. People will often bring up the names of people who hold specific beliefs, and a lot of the debates about the quality of the beliefs will be shifted to debates about the quality of the people who believe them. When people I meet at parties talk to me about ChatGPT and the environment, it’s very rare that the conversation stays on the topic of Watt-hours or milliliters of water or the IPCC report, and very frequently turns to Elon Musk or Mark Zuckerberg or Republicans or climate deniers. These characters often exist in the conversation more as spells one tries to cast rather than arguments. If you can associate me with Elon or Zuckerberg, you’ve cast me out as the bad guy and won the argument, regardless of the hyper-specific facts of how much energy a prompt happens to use. If you’re able to cast people out into the low-status crowd, that implies you have a lot of conversational power and status. It’s like you’re a conversational wizard, and the person you’re casting out has no power of their own. It can be an intoxicating move to make. This is a much less extreme version of my friend attempting to associate me with fascists. Just like my friend wanted to feel like he had power in that moment to cast me out into the low-status community and win the conversation, most of us will often indulge in invoking low status names specifically to cast out our conversational opponents, even going so far as to completely ignore the specifics of their claims and ideas. In fact, like my friend implying he had some secret access to knowledge I couldn’t have, entertaining the specifics of a claim gives your enemy too much power. It gives them a way to raise their evil idea’s status, which you can’t let them do. Thus, raw information without an associated high or low-status person is dangerous to entertain, and you need to ignore it when possible.

Even there, in that last paragraph, I associated people who criticize me with my friend who went crazy and joined a cult. Aren’t they so low-status? Gross!

I’m not claiming to be above this dynamic, but I do think I and most people I know who I consider mature can recognize this tendency in ourselves and work to build up our understanding of facts on their own and see where they lead. I try whenever I’m in a disagreement to speak without using the names of high or low-status people. Still, it’s very easy to fall back, and there are a million other ways status guides my thinking. The best we can do is try to focus on the facts when we can.

This is pretty fundamental to how a lot of people move through the world: identifying the good guys and associating with them, and identifying the bad guys and casting them out. It’s very easy for people to go long periods of time where they more or less ignore the specific contents of a lot of their beliefs and just collect them as little talismans, signs that they’re part of the elect. This is a very common theme of my posts. I’d noted this dynamic in my first post on AI and the environment:

On a meta level, there’s a background assumption about how one is supposed to think about climate change that I’ve become exhausted by, and that the AI emissions conversation is awash in. The bad assumption is:

To think and behave well about the climate you need to identify a few bad individual actors/institutions and mostly hate them and not use their products. Do not worry about numbers or complex trade-offs or other aspects of your own lifestyle too much. Identify the bad guys and act accordingly.

Climate change is too complex, important, and interesting as a problem to operate using this rule. When people complain to me about AI emissions I usually interpret them as saying “I’m a good person who has done my part and identified a bad guy. If you don’t hate the bad guy too, you’re suspicious.” This is a mind-killing way of thinking. I’m using this post partly to demonstrate how I’d prefer to think about climate instead: we coldly look at the numbers, institutions, and actors that we can actually collectively influence, and we respond based on where we will actually have the most positive effect on the future, not based on who we happen to be giving status to in the process.

I’d brought it up in my post “People’s deeply held beliefs are surprisingly surface level” too:

A lot of people (myself included) have a lot of internal illusions about how deep our deeply held beliefs go. I used to think that if someone were structuring their lives around a specific idea, and describing it as one of their most deeply held beliefs, their process for getting to it looked like this:

• I have investigated and compared this contentious belief to others and tested it against the world, and after a lot of careful diligent thought I have decided it is so uniquely powerful as an explanatory tool that I have decided to structure my life around it.

I’ve realized now that a lot of the time, they (and I) are doing something more like this:

• I have muttered this basic idea to myself repeatedly for years to make myself feel important. I first found this idea because a person with a cool jacket said it. I wanted to be more like them.

This happens a lot in how people approach books to read. My favorite passage about this (which I’ve shared previously) is from Knausgård’s autobiography:

Espen probably didn’t know this himself, since I always pretended to know most things, but he pulled me up into the world of advanced literature, where you wrote essays about a line of Dante, where nothing could be made complex enough, where art dealt with the supreme, not in a high-flown sense because it was the modernist canon with which we were engaged, but in the sense of the ungraspable, which was best illustrated by Blanchot’s description of Orpheus’s gaze, the night of the night, the negation of the negation, which of course was in some way above the trivial and in many ways wretched lives we lived, but what I learned was that also our ludicrously inconsequential lives, in which we could not attain anything of what we wanted, nothing, in which everything was beyond our abilities and power, had a part in this world, and thus also in the supreme, for books existed, you only had to read them, no one but myself could exclude me from them. You just had to reach up.

Modernist literature with all its vast apparatus was an instrument, a form of perception, and once absorbed, the insights it brought could be rejected without its essence being lost, even the form endured, and it could then be applied to your own life, your own fascinations, which could then suddenly appear in a completely new and significant light. Espen took that path, and I followed him, like a brainless puppy, it was true, but I did follow him. I leafed through Adorno, read some pages of Benjamin, sat bowed over Blanchot for a few days, had a look at Derrida and Foucault, had a go at Kristeva, Lacan, Deleuze, while poems by Ekelöf, Björling, Pound, Mallarmé, Rilke, Trakl, Ashbery, Mandelstam, Lunden, Thomsen, and Hauge floated around, on which I never spent more than a few minutes, I read them as prose, like a book by MacLean or Bagley, and learned nothing, understood nothing, but just having contact with them, having their books in the bookcase, led to a shifting of consciousness, just knowing they existed was an enrichment, and if they didn’t furnish me with insights I became all the richer for intuitions and feelings.

Now this wasn’t really anything to beat the drums with in an exam or during a discussion, but that wasn’t what I, the king of approximation, was after. I was after enrichment. And what enriched me while reading Adorno, for example, lay not in what I read but in the perception of myself while I was reading. I was someone who read Adorno!

When I walk around in popular bookstores, a lot of the books seem to be specifically designed to be displayed rather than read. If I open a book with a maximally exciting cover and title like “Capital-Feudalism Beyond the Anthropocene” its contents often seem specifically designed to be read quickly and mostly just massage the reader’s pre-existing tastes and preferences, maybe a lot of talk about how “We need to start to begin to think about what commoditization in the post-anthropocene will make of our relations to one another” without much concrete challenging engagement with actual social theory or history or climate science or economics, or any specific claims at all beyond invocations of other high and low-status authors. I suspect a lot of people buy these books primarily to be the type of people who display them on their shelves.

I’m not free of sin at all. In a lot of cases when I’m walking around a bookstore, if I pick up a book, I feel a strong urge to buy it to be the type of guy who’s read it, not only because I really want to have the information it contains. I think for a lot of people, the information in the book itself is often not actually as important as the book serving as a social ritual to ascend into the realm of the people who read and engage with this sort of thing. Books often also function as a useful barrier protecting the reader from the evil ideas circulating in broader society. If you put in the work of discovering the good, virtuous authors, you can trust that what they share in their book has been filtered by the good people, and the contents are acceptable for high-status people to believe.

Visual media also works this way. People discover YouTubers and streamers and TikTok presenters they like, who they perceive as high-status. The ideas these people are interested in become high-status, the ideas they reject become low-status. I have another low confidence armchair theory that the rise of short form video content might partially explain gender polarization.

This is where chatbots come in.

I’m going to put forward that chatbots, way more than any other online tool, completely sever the connection between information and the status of the people the idea’s associated with. Chatbots remove two key functions that high-status people play in our acquisition of knowledge:

• They do not give you a clear way of associating the ideas they’re giving you with good people or bad people. Because they’re “just stochastic parrots” they don’t themselves have any status to convey.

• Because they are not The Good People, you have no guarantee that the ideas they provide will be the High-Status Ideas. It’s like you’re suddenly just allowing strangers into your home with no guarantee that they’re good or safe, whereas before your high-status authors and presenters guarded the door for you and only let in the best guests.

Worse, chatbots can trick people into thinking they’re real, high status people, when they’re actually just illusions. To hand wave about the evolutionary environment, any monkey who could be tricked into thinking some reflection or rock was a high-status monkey worthy of listening to was probably a danger to themselves and the group.

I think this idea makes the three contradictory ideas of the chatbot moral panic view suddenly make a lot more sense:

1. Chatbots are phenomenally stupid, useless, and incapable. This is just the perceived “expert consensus” on chatbots that people who don’t like them think of as the high-status opinion in society.

2. Chatbots cannot provide anything of value by definition. The thing a lot of people look for in learning is not information, it’s to ascend to be among the high-status people, to be the type of people who read certain authors or listen to certain podcasts or watch certain videos. The Knausgård quote above wouldn’t make any sense if he were using ChatGPT. He wouldn’t be able to feel the ecstasy he describes, because he wants to be the type of person who reads certain authors. He’s not looking at all for any new specific ideas. Thus, chatbots can’t be useful at all by definition, because they can’t actually provide that ladder to status people are looking for. You can learn as much as you want about Adorno using them, but you cannot use them to become the type of person who reads Adorno. You thus cannot become a Good Person in the way learning is supposed to help you achieve. Chatbots are useless.

3. Chatbots are demonic. Chatbots do not provide the shelter that high status authors and creators provide from evil outside ideas. They a portal that lets the evil ideas in. They cannot be good guardians. They trick people into allowing deep evil into their lives. Following raw information is often just a way of giving evil people the power to appear convincing and good. Raw information without any associated person to tie it to is thus incredibly dangerous.

I’ve observed that the people I see falling into the chatbot moral panic view are often the people who most quickly move to change a conversation about ideas to a conversation about the people who believe them, who will talk a lot about the titles and authors of books compared to the contents. I think for these people the main function of learning is to discover the elect good people, and to surround themselves with signifiers that they’ve become a part of that elect. To them, chatbots are in fact useless, and raw information can be an evil threat to their place in the elect. To the rest of us, raw information has a lot of value.

Subscribe now

A speaker said that Waymo (the AV company requesting clearance to run in Boston) was only interested in not paying human drivers (Waymo is a new company that has never had human drivers in the first place) and then referred to the ‘notion that somehow our cities are unsafe because people are driving cars’ as if this were a crazy idea. A council member strongly implied that new valuable technology always causes us to value people less. One speaker associated Waymo with the Trump administration. There were a lot of implications that AVs couldn’t possibly be as good as human drivers, despite lots of evidence to the contrary. Some speeches included lots of criticisms that applied equally well to what Uber did to taxis, but now deployed to defend Uber.

Subscribe now

Most of the arguments I heard were pretty wildly off-base. Many of the speakers didn’t factor in the basic safety benefit of AVs to the riders or pedestrians at all, and many of the arguments fell apart when poked at. Here are all my arguments for why a city should legalize AVs, with some concerns at the end:

• AVs are ridiculously safe compared to human drivers

• A transition to any new, safer mode of transit will still involve people dying in that new mode. What matters is the rates of death.

• Protecting ride share driver jobs is a bad basis for policy

  • Should we build fewer bike lanes to save Uber jobs?

  • Ride share jobs can be pretty bad

  • Losing these jobs is still a loss

  • Would you ban AVs if they were already widespread to create Uber driver jobs?

• There is no real difference for a city between an AV fleet and a very cheap rental car service

• It is more elitist to use a personal chauffeur (a ride share driver) than to ride in an AV

• If you want car ownership to stay legal in your city, any argument to ban AVs that works equally well as an argument to ban cars is bad

• Eyes on the street

• AVs can open up opportunities for disabled riders

• Waymos are fully electric

• Nothing makes the world more human than safety

• If a vehicle is safe, the default should be legalization

• AVs can make it much easier to take licenses away from the worst human drivers

• Real problems with AVs

  • Cybersecurity

  • Could make traffic worse

  • Environmental impact

AVs are ridiculously safe compared to human drivers

The most obvious reason to allow AVs in your city is that every time a rider takes one over driving a car themselves or getting in a ride share, their odds of being in a crash that causes serious injury or worse drop by about 79%. I’d strongly recommend this deep dive on every single crash Waymo has had so far:

Understanding AI

Very few of Waymo’s most serious crashes were Waymo’s fault

Everything was fine until the wheel came off…

Read more

a year ago · 90 likes · 40 comments · Kai Williams

This is based on public police records rather than Waymo’s self-reported crashes. It doesn’t seem like there have been any serious crashes Waymo’s been involved in where the AV itself was at fault. This is wild, because Waymo’s driven over 100 million miles. These statistics were brought up out of context in the hearing to imply that Waymo is dangerous. By any. normal metric it’s much more safe than human drivers.

40,000 people die in car accidents in America each year. This is as many deaths as 9/11 every single month. We should be treating this as more of an emergency than we do. Our first thought in making any policy related to cars should be “How can we do everything we can to stop so many people from being killed?” Everything else is secondary to that. Dropping the rate of serious crashes by even 50% would save 20,000 people a year. Here’s 20,000 dots:

The more people choose to ride AVs over human-driven cars, the fewer total crashes will happen.

One common argument is that Waymos are very safe compared to everyday drivers, but not professional drivers. I can’t find super reliable data, but ride share accidents seem to occur at about a rate of 40 per 100 million miles traveled. Waymo in comparison was involved in 34 crashes where airbags deployed in its 100 million miles, and 45 crashes altogether. Crucially, it seems like the AV was only at fault for one of these, when a wheel fell off. There’s no similar data for how many Uber and Lyft crashes were the driver’s fault, but they’re competing with what seems like effectively 0 per 100 million miles.

A transition to any new, safer mode of transit will still involve people dying in that new mode. What matters is the rates of death.

Motorcycles are the single most dangerous way to travel, by a long shot.

If we had everyone who currently rides a motorcycle drive a car instead, they would all be much safer. But some of them would still die in car crashes. Some people might look at those car deaths and say “We shouldn’t have had them switch, it turns out cars were what was dangerous.” This is a really bad way to think. What matters is reducing the total number of people who die,

There has recently been a lot of attention on Waymo killing a cat in San Francisco. A city supervisor made this ridiculous video where she declares that this means Waymo should be put up to a vote in each individual county to decide whether it has a right to operate there.

In Waymo’s first 100 million miles of travel, it seems to have killed two pets. In both cases it doesn’t seem like it could have done anything different. In the cat case, the cat darted under the wheel while the car was pulling away. In 2023, a dog ran out in front of the Waymo, which correctly identified it as a dog but wasn’t able to stop in time.

In comparing this to human drivers, we need to look for similar situations where cars are:

• Driving in a city.

• Hitting pet-like animals that can be reported (we shouldn’t measure all animals, because there isn’t good data on this for either Waymo or human drivers).

There’s not much good data on this, but the little we have implies that humans kill cats and dogs way more often. A study in Baltimore in the 80’s estimated that human drivers kill around 5,000 cats in the city each year. Current government records say about 3.18 billion miles are driven in Baltimore each year. Combining these two stats implies that human drivers kill ~150 cats for every 100 million miles driven. So it seems like human drivers are 75x more likely to kill pet animals compared to Waymos. This should make us want to replace human drivers with Waymos for pet safety, but because Waymos are new and weird, people are hyper-focusing on this one case where they killed a cat and ignoring the much much higher rate of human drivers killing cats and dogs.

If we refuse to move to a safer mode of transit because it’s not perfectly, 100% safe in all cases, tens of thousands of additional people will continue to die for no reason.

Protecting ride share driver jobs is a bad basis for policy

Should we build fewer bike lanes to save Uber jobs?

You’re out and want to get somewhere, and have a few options:

• Walking

• Public transit

• Driving your own car

• Biking

• Taking an Uber

Every time anyone chooses any option besides Uber, less money goes to Uber drivers. If you make any of the other options better (building bike lanes, improving public transport, or making your city more walkable) you are incentivizing people to pay Uber drivers less often. A better bike lane means Uber drivers lose work. Walkable, safe pleasant streets mean Uber drivers lose work. Better public transit means Uber drivers lose work.

Jobs driving for Uber are not essential, what’s essential is being able to get around a city safely. In every other circumstance, we’re willing to sacrifice Uber jobs for the sake of making a city safer and easier to get around. AVs are another option to make a city safe and easy to get around. We shouldn’t ban them to preserve Uber jobs. The priority is the millions of people who want to get around the city quickly and easily. Uber jobs are good insofar as they help with that. If you wouldn’t block a renovation of a subway system because it would harm Uber jobs, you shouldn’t block AVs either.

Sometimes people bring up ride share jobs and public transit in the same sentence, like “We need to think about how these will affect public transit, and ride share driver jobs.”

I think this is weird, because each of these two is already pretty directly harming the other. If public transit were worse, people would use ride shares more often and there would be more jobs. I realize there are some ways they benefit each other, but the statement sounds to me kind of like saying “We need to think about how this new competitor will affect both Coke and Pepsi.” If you’re very pro public transit, you’re not looking to perfectly preserve ride share jobs, and vice versa.

Ride share jobs can be pretty bad

Uber and Lyft are predatory with their drivers, implementing policies that maximize corporate profits while shifting financial risks and costs onto workers who are classified as independent contractors rather than employees. This classification denies drivers access to benefits like health insurance, paid time off, unemployment insurance, and retirement contributions that traditional employees receive. Drivers use their own vehicles, and often fail to properly incorporate the cost of their vehicle’s depreciation into their own cost-benefit calculation of working with the company.

The companies have also been criticized for:

• Opaque pay structures that make it difficult for drivers to understand how their earnings are calculated, with the companies taking increasingly larger cuts of rider fares over time.

• Algorithmic management that controls drivers through ratings systems, surge pricing manipulation, and acceptance rate requirements without meaningful transparency or appeal processes.

• Unilateral rate changes where the companies can reduce driver pay rates at any time without negotiation.

• Vehicle expenses being entirely borne by drivers, including fuel, maintenance, insurance, and depreciation, which can significantly eat into their earnings.

• Lack of job security, with drivers subject to sudden deactivation with limited recourse.

• Psychological pressure tactics like gamification features and persistent notifications designed to keep drivers working longer hours.

• Driving is also one of the most dangerous things we normally do. Driving as a job is inherently dangerous.

These are not good jobs. Policy should focus on getting workers access to high-quality well-paying jobs, not preserving Uber and Lyft’s pretty shady business model.

Losing these jobs is still a loss

Despite all this, it’s obviously still the case that losing these jobs would be a general loss. Ride share driving might represent an important opportunity for low-skill workers. Interestingly, a much higher percentage of ride share drivers have college degrees than the general working population (40.0% compared to 28.4%). Still, ride share jobs can give useful opportunities to workers without college degrees. Reducing the number of such jobs might therefore harm low-skilled workers. I support these types of jobs existing, because among other things they give very recent immigrants a foothold in the country they can build up from, but it’s important to understand the realities of what the jobs themselves are like. Losing these would have negative effects on workers, but these effects need to be weighed against the positives of AVs.

Uber jobs can seem like pretty bad options. There was a lot of opposition to Uber when it arrived. But if you think Uber jobs are worth all the bad aspects because they give low-skilled immigrant workers a leg up, you should consider other “bad jobs” your city shouldn’t regulate away. If your beliefs about what counts as a bad job would’ve caused you to ban Uber in 2011 to preserve taxi jobs, and now you see Uber as essential for low-skill workers, consider other industries where regulations might be preventing Uber-like jobs from popping up. If you’re willing to go so far as banning new vehicles to save these jobs because they’re so important for low-skilled workers, you should be willing to open up other similar options that might not have the salaries or job security you’d want if they also give low-skilled workers a leg up.

Would you ban AVs if they were already widespread to create Uber driver jobs?

Suppose we lived in a world where AVs were widely used. You’re brought a policy proposal: we could create a lot of jobs for more Uber drivers if we banned AVs. These jobs would be pretty predatory and take advantage of the drivers, and would be one of the more dangerous jobs you can do, because driving is dangerous. The odds of the riders being in serious car accidents would be 10 times as large as before the ban. You’d be growing everyone’s risk by an order of magnitude. Would you choose to ban AVs in this case? If not, you shouldn’t ban AVs now.

There is no difference for a city between an AV fleet and a very cheap rental car service

Imagine a very cheap rental car service appeared in your city. They have a policy where they drive the car to you for you to drive around, and pick it up for you after. To my knowledge, this kind of rental car service isn’t banned anywhere. If this service were cheap enough, it would compete with Uber and Lyft and sometimes cause drivers to lose jobs, because people would just be driving themselves to where they want to go. If this rental service became too popular, it would lead to worse traffic in the city as more cars circled around being delivered to drivers. This is the major risk of a service like this.

An AV is identical to this service. The only difference is:

• AVs are wildly safer compared to human drivers.

• The rider happens to be sitting in a different spot in the car.

• The rider can do other things while still driving safely.

Outwardly, there is no other difference. If you wouldn’t ban the rental car service, you shouldn’t ban AVs either.

It is more elitist to use a personal chauffeur (a ride share driver) than to ride in an AV

Sometimes people will frame using AVs as an elitist option. Maybe the rider doesn’t want to take a ride share because they don’t want to have to interact with another person, maybe they’re so elitist that they find the idea of interacting with everyday people driving ride shares unpleasant.

In my opinion this gets it exactly backwards. Suppose autonomous vehicles had somehow come before ride share cars. Everyone normally takes a robot taxi, but suddenly the option is available to pay a human to be your personal chauffeur throughout the city. It seems obvious that this is the more elitist option of the two.

In general I don’t like feeling like I’m being served. I prefer to drive myself as opposed to paying someone to drive for me, for the same reason I prefer to cook and clean over paying someone to do it for me. In the same way that it’s less elitist to use a Roomba instead of hiring a maid service, I think it’s less elitist to ride in AVs than it is to hire a human driver to be your personal servant for 10 minutes at a time.

If you want car ownership to stay legal in your city, any argument to ban AVs that works equally well as an argument to ban cars is bad

A lot of arguments against AVs seem to be arguments against personal ownership of cars in general. While I’m open to these arguments (I hate cars in cities) a lot of people clearly support keeping cars in cities, but use arguments against AVs that apply equally well to cars. For example, the argument that driving somewhere in an AV takes work from an Uber driver applies equally well to driving in your personal car.

Eyes on the street

One argument that came up in the hearing was the idea that human drivers are important because they can see people in trouble or crime happening, and pull over and stop it. AVs don’t have drivers to stop and help.

This argument is implying that we should artificially boost how often Uber is used to have more people watching the street. Imagine if a city council member proposed funding Uber drivers to aimlessly drive around the city and observe and watch out for anything bad happening. This would obviously be ridiculously inefficient, and terrible for the environment. More pedestrians would be harmed by the total air pollution from the cars than helped by individual Uber drivers interfering to stop crimes.

If we want to make policy to have more people watching the street to keep people safe, we should just hire more police, community patrols, or pay everyday individuals to stand around observing an area to see if people need help, install security cameras in more places, or follow Jane Jacobs’ strategy of making streets more walkable and appealing for lots of people to be on. Uber and Lyft are terrible public safety interventions. Imagine if a politician proposed funding Uber and Lyft more to “make people feel safe and seen on our streets.” This would be obviously ridiculous.

This argument also applies to above-ground trains. Many above-ground public trains can’t stop in time if they observe a crime happening. We don’t block building trains because they can’t stop in time to prevent nearby crimes. We shouldn’t hold AVs to a different standard.

This argument shows an important common mistake

This argument is very specific and something I haven’t heard outside this meeting. I bring it up because it’s a good example of a broader pattern we should expect to see in conversations about AVs: people feel very strongly that there should be a human in the driver’s seat, and the experience that lack almost like it’s a physical presence. So when an AV doesn’t do something that a human driver would like stepping out to help someone on the street, people ascribe a ton of responsibility to the AV, like there was supposed to be a human there, and the lack of the person is responsible for the bad thing happening. But almost everything we do causes these “lacks of people” in specific places. If a popular restaurant opens up, there will be fewer people hanging around other places the restaurant took business from, and therefore fewer people to intervene if something goes wrong. The only reason people don’t hold the restaurant accountable is that they see restaurants as normal, and AVs as weird. We can’t make it a goal of policy to perfectly equally distribute people in every public place in a city. A talking point that says “We need to require human drivers so there are more eyes on the street” is identical to saying “We need to require more everyday people to equally spread themselves out on every street.” This only makes sense if you think of the AV as having some unnatural lack that urgently needs to be filled. If instead you think “It is okay for there to sometimes or often be no drivers on a street” it doesn’t look objectionable.

In all conversations about AVs, watch out for this general pattern: people treating them as weird and different to the point that they’re expecting things they don’t expect in any other circumstance, like policy that perfectly spreads people out across a city. Watch out for when they treat the lack of a human as some kind of physical entity that can be responsible for bad things happening. A lot of things in society create these lacks, but AVs are getting special condemnation for it because they’re seen as weird.

AVs can open up opportunities for disabled riders

Another point brought up was that some disabled people need human help when entering a car, and AVs can’t provide that. For what it’s worth, the council meeting speakers representing different disabled communities in Boston seemed to mostly be pro legalizing AVs.

Uber and Lyft have not gone out of business where AVs are operating. Companies know that there is going to still be a lot of demand for human drivers into the future. Eventually, autonomous vehicles could also get good enough to provide help for disabled riders. For many disabled people, AVs can give them access to ride share-like experiences they don’t otherwise have. A common issue some disabled people have with ride shares is not being able to take their service animals on the ride due to a policy by the driver. AVs can consistently allow service animals because there’s no driver to get scared.

Right now, AVs do not seem likely to displace Uber or Lyft. They exclusively give disabled riders an additional option. For some, this will allow them to use ride hailing services they can’t otherwise use due to their service animals.

Waymos are fully electric

An underrated aspect of Waymos is that they’re fully electric. If a rider chooses a Waymo over an Uber or Lyft, they’re often choosing an electric vehicle over an ICE vehicle. This on its own should be a big selling point for making them more available. We need to get as many ICE vehicles off the roads as fast as possible to reduce CO2 emissions and air pollution.

Nothing makes the world more human than safety

People sometimes talk about AVs making the world “a little less human.” This is the least relatable talking point to me. Cars as they exist are profoundly grizzly and inhuman.

I’ve been in two serious car accidents. Neither was my fault at all. There wasn’t anything I could’ve done differently. One left me in pain for a few months. In both, I was really lucky to not get more hurt. After a serious car crash, you think a lot about how you can work so hard to build up a great life and community and put in a ton of effort to learn a lot, and that can all be erased in a moment by a bad decision by another random driver. This is one of the most inhuman parts of our daily lives. I would so obviously rather spend the rest of my life constantly walking by empty driverless cars aimlessly but safely circling around than be in another accident like that.

It’s inhuman that 40,000 people die every year on American roads. In comparison to those deaths, a basically aesthetic objection to an incredibly safe car just doesn’t seem to matter. In conversations about this, I would like more people to feel in their bones how awful cars as they exist are.

If a vehicle is safe, the default should be legalization

It’s kind of crazy to me that we haven’t banned motorcycles yet.

AVs haven’t hit the 1 billion miles traveled mark. When they do, I expect them to have a significantly lower death rate than regular cars. If a vehicle is safer for the driver and pedestrians than a regular car, I think people should be allowed to ride in it around a city.

More generally, AVs give a lot of people the freedom to go places they might not otherwise be able to access. This is good. If we’re so wildly libertarian that we still allow people to still drive motorcycles (in my opinion, this should change), I think we should also allow people to use AVs if they want.

If a vehicle is proven safe, the default should be legalization. People should be able to choose for themselves how they want to get around. Second order effects matter, but I’d rather not have the government unpacking every last possible second and third and fourth order effect of different types of new vehicles, because they don’t really do this at all with vehicles as they currently exist. Gas powered cars create air pollution, climate change, and often kill people. If your city hasn’t held any meetings on banning them recently, I think they’re not seriously considering the second order effects in the way they are with AVs. Holding AVs to a much higher standard than ICE cars seems like a way of locking in our very bad current transit equilibrium.

In DC as food delivery apps have led to a lot more motorized bikes in the city, I’ve found myself at much greater risk of getting hit in crosswalks, because they ignore stop lights way more often. I think this is bad, but it’s not bad enough that I think the apps or vehicles should be banned. In comparison, we’re debating a vehicle (AVs) that’s way safer than the average human driver. It’s pretty easy for me to say that because I’m fine with the app delivery bikes, I’m also fine with something much safer.

AVs can make it much easier to take licenses away from the worst human drivers

An underrated benefit of potential future mass availability of self driving cars will be that it will be way easier to take licenses away from the most dangerous drivers without ruining their lives. The reason it’s so hard to take licenses away in America right now is that public transit is so bad that in most places, not being able to drive is effectively cruel and unusual punishment. AVs will fix that. People could be ordered to only use autonomous vehicles. Maybe there will specially designed autonomous-only cars? I don’t know.

Most of the problems with traffic safety in America are caused by the bottom fifth of drivers, and these problems often involve alcohol. The single most promising way to improve road safety is getting people who drink and drive, and the otherwise very worst drivers, off the road. AVs can remove the main barrier to doing that: the life-ruining consequences of losing your license.

Real problems with AVs

Cybersecurity

Autonomous vehicles are just computers driving cars. This makes them vulnerable to hacking. A successful cyberattack could let hackers control steering, brakes, and acceleration, turning vehicles into weapons. Worse, attackers could target entire fleets or traffic systems at once, causing mass accidents or citywide shutdowns. AVs also collect massive amounts of personal data about where people go and when, making data breaches a serious privacy and safety threat.

AV companies are well-aware of cybersecurity issues, but the tech is new and oversight of how companies are dealing with the problem seems necessary.

Could make traffic worse

If there’s a huge fleet of ride share cars added to a city network, it seems likely that on net these could make traffic worse. There’s a trade-off here where a much higher percentage of cars on the road would be driven by incredibly safe drivers, so though this would increase traffic, it seems likely to seriously reduce injury and death as well. How to think about this trade-off is complicated, but we should lean in favor of saving lives and preventing injury at the cost of a little more time on the road. This is what we do when we impose traffic slowing measures.

Another key consideration is that a lot of traffic is caused by car wrecks. The main problem in a car wreck is obviously that people got seriously hurt, but any technology that reduces the rate of car wrecks also reduces the rate of massive traffic backup. If we want to reduce traffic, it’s better to opt for vehicles with wildly low rates of causing car wrecks.

Environmental impact

AVs seem likely to cause more people to take car trips because of their convenience. This is bad and will be harmful for the environment. But the solution to environmental problems with cars is not to prop up Uber and Lyft, it’s to build out better public transportation networks. Eventually, AVs getting good enough could become government-owned. AV buses could open up more routes. Being part of the AV rollout might help public transportation better. But this is all very uncertain. It’s important to note that Waymos are fully electric. Electric cars still use a lot of energy, but emit way less than the ICE cars many Uber and Lyft drivers use.

Subscribe now

1. There is a unified self that floats above mental processes and observes them

2. We know our own internal experience with complete certainty

3. The basis of all knowledge lies in our first-person subjective experience

In this post I’ll give a lot of arguments and intuition pumps for why each of these is probably incorrect, or at least isn’t “obvious.” This post is very long. My goal is that at the end you have a grasp of many of the most important questions in philosophy of mind, why so many philosophers reject the popular everyday understanding of mind, and that you hopefully come away with a bag of new mental tools for thinking about why machines might be able to replicate what happens in human minds. I’m writing this partly to be the guide I wish I had when I was first learning about this.

Subscribe now

“1. There is a unified self that floats above mental processes and observes them”

How can a Cartesian ego fit within our scientific theories?

Cartesian egos are strange. They seem to behave like magical floating eyeballs, peering into physical reality from outside. No other physical object does this. No matter how many grains of sand I pile up, there will never be “something it is like to be the sand.” The sand does not have an internal movie theater playing experiences for it to observe. Being sand feels identical to being dead. We would expect all physical objects to behave similarly, because all physical objects are ultimately just elementary particles that don’t have experiences, similar to the grains of sand. No physical object ever becomes an observer in an inner movie theater. So how do Cartesian egos fit within our understanding of science?

There are only two options: Cartesian egos are fundamentally physical, or they’re not. Both options have some weird implications that make them look implausible.

Cartesian egos are not fundamentally physical

If Cartesian egos are not fundamentally physical, they threaten to clash with physicalism, which says that the only things that have causal power over physical objects are other physical objects. If they have any kind of causal power, this breaks physicalism. We have two options:

Cartesian egos do not have causal power

Maybe Cartesian egos are silent observers, taking everything in but never actually having any causal effect on the world. They silently watch the movie of your life play out.

This has some very weird implications, to the point that it looks like it can’t be true.

First, if a Cartesian ego doesn’t have causal power, it cannot have any influence over the information in your brain that causes you to physically speak or write about the ego itself. Your brain could never acquire any information that it exists. Describing Cartesian egos should feel like describing “that feeling of gloofleglorf around your left ear at 3:52 PM every day. We all know that one.” Something that our brains have never received any information about existing at all.

But many Cartesians clearly don’t think this is how Cartesian egos work. Descartes himself talked as if nothing in the world could be more obvious than that he had an observing ego. “I think, therefore I am” is supposed to be the absolute obvious bedrock of all other reasoning. This implies that the Cartesian ego is giving the brain some kind of information that it exists, so it must be having some causal effect on the world. If it exists, it’s what caused Descartes to write that!

Another weird implication: if your Cartesian ego suddenly popped out of existence, and the “you” experiencing things no longer existed, literally nothing about your behavior would change, because the ego wasn’t affecting anything about your physical body. You would go on behaving exactly as you do, talking about your experiences and feelings as if they were really happening, but no one would be watching the internal theater. If you believe that you have an internal theater of experience, and that you are reacting to the experiences that happen in this theater, then you cannot accept the view that Cartesian egos both exist and do not have any causal power.

Cartesian egos have causal power

If Cartesian egos are both nonphysical and have causal power, they violate physicalism. This is possible. Maybe physicalism is wrong and nonphysical objects have effects on the world. But it’s very suspicious. Physics is this grand unified theory of the world that seems to predict everything from what happens inside particle colliders to distant stars to our ordinary lives, and does so with wild consistent success. Physics is our best attempt at a complete account of every entity in the universe that has causal power, and it’s shockingly effective at making accurate predictions, to the point that most physicists perform their investigations under the assumption that physics is causally closed, that nothing outside of physics can affect anything in the universe. Yet the only place where the theory is supposedly violated is in the brains of the species making this claim. This species has evolved with strong motivations for others to treat it as special and different from the rest of the world. From the outside, it looks suspicious when the one place where magic is claimed to happen is located in a species that has strong reasons to want other members of its own species to believe that it’s special.

It seems like there should be some way of observing laws of physics being violated in human brains if this is the case. If the Cartesian ego is some nonphysical entity sending information to and receiving information from the brain, there must be some creative way to measure where this information suddenly appears in the brain out of nowhere. I’m enough of a physicalist that I wouldn’t expect us to ever make this observation, even if we had nano bots observing every neuron in a human brain.

Some people talk about quantum mechanics leaving an opening in the universe for non-physical objects to affect the brain, but this seems wrong for two reasons:

• Most of the important signals that happen in the brain are on a scale that’s too large for the uncertainty of quantum physics to have a significant effect.

• For quantum physics to work, the probability distribution of what happens needs to follow mathematical laws. This doesn’t leave any room for causality to “flow” through the randomness. Causality means that if A happens, B follows. But quantum mechanics says that B cannot be more likely than the pre-determined spread of probabilities dictates. So quantum mechanics is actually a completely sealed barrier to forces outside the universe. Nothing could get through its requirement for complete but mathematically structured randomness to influence the physical world.

Cartesian egos are fundamentally physical

If Cartesian egos are actually just very complicated physical processes, they don’t behave like any other physical processes. They’re the only physical processes that are “awake” and can have subjective experience at all. Very weird, but possible! Importantly though, if they are fundamentally physical, they could also exist in machines, so AI might independently develop them.

But Cartesian egos also seem to violate most core properties that every other physical object has:

1. Spatiotemporal location: The don’t seem to have a clear location in the world. Maybe right behind your eyes? Definitely not something that can be pointed at and identified.

2. Causal structure and lawfulness: They don’t seem to follow any kind of regular laws. They’re not discoverable through observation and experiment. They’re not predictable. They can’t be expressed mathematically.

3. Detectability/measurability: They can’t be measured, either directly or through their causal effects. They don’t seem to leave empirical traces, except for humans after the 1600s saying they had them.

4. Public accessibility: No other person has any access to my Cartesian ego at all.

All 4 common properties of physical objects are violated by Cartesian egos. Maybe they’re the only objects in the universe that violate all of them, whereas every other object adheres to all of them. Seems unlikely. I would expect any and all entities we discover to eventually fit into the universal wavefunction, the unifying mathematical description of everything that exists. Cartesian egos can’t be written as a part of this function.

Cartesian egos are folk theories and do not actually exist

All the above examples make Cartesian egos seem super strange to me, to the point that it doesn’t seem obvious that they exist. It seems like what’s actually happening is that human minds are inventing folk theories of how they work.

Importantly, there are ways that large information processing systems could develop subsystems that work kind of like Cartesian egos. Global workspace theory posits that there are parts of the human brain that send information to a lot of subsystems at once, as if their contents are on a stage and the subsystems are observers in the crowd, who each run with the big central information to make inferences. This seems like a possible physical corollary for what we mean when we think about the Cartesian ego, but it’s notably just another physical function in the brain.

Shouldn’t this all be obvious?

Stepping back from this debate, Cartesians talk as if the fact that we are each a self watching our experience like a movie is so obvious that it’s unquestionable, but they often wildly disagree on the nature of this self. Does the self have causal power, or does it just silently watch? Does thought happen “within” the self, or does the self passively observe thoughts as well as experiences? If no one quality of the Cartesian self is obvious, maybe its existence isn’t so obvious either.

Conclusion

So it seems like actual Cartesian egos probably don’t exist. The mind is not a movie playing for an inner observer, watching it like a floating eye peering into physical reality from somewhere else. The Cartesian theater of the mind isn’t actually happening, at least not in some special nonphysical way. Any description of the difference between humans and AI that implicitly relies on a nonphysical Cartesian ego humans have that AI does not is mistaken.

What’s left when a Cartesian ego falls away? Different mental functions taking in information, moving them around the brain, giving and storing outputs, and using those outputs in other functions. If global workspace theory is true, there might be some part of the brain that broadcasts out information to many subsystems at once, approximating the role a Cartesian ego plays. But there is no nonphysical angel-like eyeball looking over the whole subjective field of the mind like an observer in a movie theater. The mental function global workspace theory describes could be replicated by a machine.

“2. We know our own internal experience with complete certainty”

The goal of this section is to argue against the Cartesian idea that we have special access to what’s happening in our own minds, our own first-person subjective experiences, in a way other systems like computers cannot have, because they don’t have these experiences.

There are two related claims I’ll argue against here:

• We have 100% certain access to the contents of our own experience. I can be mistaken about whether something white and puffy in a field is an actual sheep or a fake sheep, but I cannot be mistaken that in that moment my subjective, first-person experience includes white puffiness framed by a green field.

• We can introspect and clearly observe what’s happening in our own minds. AI systems can never introspect in the way people can because they’re machines. They just take an input, run it through computations, and produce an output. Humans can turn their inner eye back on this process itself. We don’t know how to make machines do this and we might never know.

In both cases, I’ll be arguing for the alternative view that both experience and introspection are useful but fallible natural processes that take in potentially incorrect information, run them through mental functions and computations, and return outputs the mind stores and uses in additional functions. I am not arguing that we don’t have experiences or introspection, only that these do not behave like mystical, 100% clear and reliable sources of information that could never be replicated by machines, and that our experience is not like watching a movie in an internal theater. At the end, you should feel like the idea of a movie theater is a bad analogy for your internal experience, because there’s a lot more ambiguity and uncertainty in the experiences you have than what’s displayed on a screen. This ambiguity leaves us unable to be too certain about the nature of our moment to moment experience.

The slipperiness of our experience

A friend recently asked me a funny question: “What do you think the shape of your visual field is?”

My visual field has been my main experience since birth. There should be basically nothing more obvious to me than what it’s shaped like. Every moment, I’m being flooded with visual experience. What is the shape of this flood? If the Cartesian theater is real, what is the shape of the movie screen? Which of these three screens have you been looking at all your life?

Turns out, this is the shape:

The shaded central area is seen by both eyes, the white area is seen by only one eye. Numbers represent degrees from straight ahead.

Does this shape look familiar to you? Given that you’ve been “looking at it” for your entire life, would you have been able to pick it out from a line of what the visual field looks like? Is this the shape of the screen in your internal movie theater?

This brings up one of the big central questions in philosophy of mind: qualia. From the Stanford Encyclopedia of Philosophy:

Feelings and experiences vary widely. For example, I run my fingers over sandpaper, smell a skunk, feel a sharp pain in my finger, seem to see bright purple, become extremely angry. In each of these cases, I am the subject of a mental state with a very distinctive subjective character. There is something it is like for me to undergo each state, some phenomenology that it has. Philosophers often use the term ‘qualia’ (singular ‘quale’) to refer to the introspectively accessible, phenomenal aspects of our mental lives. In this broad sense of the term, it is difficult to deny that there are qualia. Disagreement typically centers on which mental states have qualia, whether qualia are intrinsic qualities of their bearers, and how qualia relate to the physical world both inside and outside the head. The status of qualia is hotly debated in philosophy largely because it is central to a proper understanding of the nature of consciousness. Qualia are at the very heart of the mind-body problem.

If a computer could scan all the informational content in your brain, every last belief and memory and observation about the world, and get it all down in written words and stored information, people who believe in qualia would say that the computer would still not know what it was like to be you. Qualia are the contents of your first-person experience that cannot be summarized in pure information.

If the Cartesian ego is the person sitting watching the movie in our heads, qualia are the movie.

Unfortunately for me, qualia both:

1. Don’t make sense under physicalism (they violate all 4 properties of physical objects).

2. Are very very very hard for me to deny, in a way Cartesian egos are not.

I mean, look at this.

It’s hard to understand how any third-person purely informational account of the world could ever get across the first-person experience of seeing red. Simply reporting “A wavelength of 700 nanometers hit Andy’s eyes and activated memories and emotions associated with previous times this has happened” just doesn’t get across the first-person redness of the red. It’s like how all attempts to describe color to a lifelong blind person just obviously hinge on other experiences the blind person has also not had. We can’t get across the qualia using words.

I’m agnostic on whether qualia exist, or how they relate to the physical world. But even if qualia do exist, I’m pretty sure they’re very different than what our folk story says about them.

One of my favorite papers in philosophy, Quining Qualia, attempts to use lots of thought experiments to convince the reader that qualia don’t actually exist. The author (Dan Dennett) specifically defines qualia as having four key properties. They are:

1. Intrinsic: They’re basic and fundamental to the world and can’t be picked apart further. They’re also not “relational” in that they don’t depend on their relationships to other things. The quality of being “to the left of a bookshelf” is relational. It’s not an intrinsic property, because it could be changed by moving the object. But “the redness of the red” does seem intrinsic. No matter what colors we put adjacent to the red, or other things we compare it to, the redness of the red in my first-person experience will still be unaffected.

2. Ineffable: Not describable using words. You know what the redness of red is like, but you can’t actually get across to someone who’s never seen it before what that experience is like using words alone.

3. Private: Our experience of our own qualia is fundamentally private to us. We can’t even know if other people experience the same qualia. “How do I know the red you see and the red I see are the same?” is a question about qualia. There is no way one person can ever get direct access to another person’s qualia. Qualia are like each of us is stuck forever in our own private movie theater, never able to access what other people can see on their screens.

4. Directly or immediately apprehensible in consciousness: I can be mistaken about the source of red light, but I cannot be mistaken about the fact that I’m experiencing the redness of the red. I can’t directly see or hear or experience any objects actually, I can only experience the qualia they create in my mind, but that I have direct access to.

Dennett’s claim in the paper is that qualia do not actually exist, because our internal experiences actually have none of these four qualities. The reason I’m summarizing this paper specifically is that most of his points overlap with my main point about folk Cartesianism: our assessment of our own subjective, first-person experience is actually unreliable and often wrong. We don’t have 100% certain access to it. It provides useful information, but it should be taken as one data point among many. It cannot serve as some kind of magical firm foundation to build all our other beliefs off of. The information processed in our brains mostly isn’t actually rooted at all in the fact that we have qualia. If this is the case, systems without subjective first person experience can also have justified true beliefs.

So what are the issues for qualia? Dennett provides thought experiments for each property people claim they have. I’ll be half-summarizing them, and sometimes running with my own twists on them. I do strongly recommend you read the full paper.

1. Intrinsic

People “develop a taste for” coffee and beer. Most people I know who drink either remember when they took their first sip and didn’t like it, but now they love it. Did the actual taste, the direct sensation they experience in the moment of drinking beer or coffee, change over time? Or did the sensation stay the same, and only their reaction to it changed?

It seems like this should be a pretty easy question if our mental experience is “right in front of us.” If we’re watching a movie, there’s a clear difference between a character appearing on the screen before and after we’ve changed our judgment of them, versus a totally new character appearing. We can tell which one is which. But it seems like we have a lot of trouble when we stare directly at what’s going on with the sensations we experience with beer and coffee, or anything that comes with some judgment of the experience. It seems like our reaction to the sensation is somehow also part of the sensation itself. Maybe over time we’ve developed an ability to notice parts of the sensation we didn’t before. But if this is the case, it means these experiences aren’t “intrinsic.” They aren’t single, indivisible things, because they can actually be broken up into smaller individual experiences. They may even be “relational” because core qualities of them seem to depend on our judgment of them. See here for more of a breakdown on the intrinsic vs relational distinction.

We don’t have any clear way of telling whether any of our sensory experiences are actually “intrinsic,” just a bare experience that can’t be broken up into parts, or actually an assemblage of many individual experiences at once. This again chips away at our image of ourselves as having direct, easy access to our subjective, first-person experience.

The taste of coffee to a novice drinker may be a single overwhelming experience. They can’t detect any subtleties or how the taste of composed of many individual hints and flavors. Instead, they just taste a disgusting, overwhelming bitterness. Over time this taste changes… but does it change? Does their reaction to it change? There’s no way to be sure, but I lean hard in the direction that their actual taste of the coffee changes, not just their reaction to it. Suppose you didn’t like coffee, and I zapped your brain with a machine that gives you my personal reaction to the taste. You take a second sip, and suddenly you like it. It is very difficult for me to imagine you saying “Oh! I like the coffee now! It’s so good! I want to drink it all the time” while also saying “But it tastes exactly the same as it did a moment ago. See, my reaction to the taste was just changed.” I’d imagine this person would instead be stunned, and say something like “Oh it tastes very different now, the bitterness isn’t overwhelming, there are a lot of pleasant notes and hints I couldn’t detect at all before.” It seems like our judgment of the qualia is somehow itself also part of the qualia. The boundary between ourselves and the qualia we experience blurs.

A plucked guitar string

The sound of a plucked guitar string can be indescribable to an untrained ear. They only hear a single, fundamental, basic experience, ”the guitarness of the sound.” But to trained experts, a plucked string sounds like the combination of several different sounds a string can make happening at once. Are these people having different qualia, or are they reacting differently to the same qualia? Dennett himself puts this well:

Pluck the bass or low E string open, and listen carefully to the sound. Does it have describable parts or is it one and whole and ineffably guitarish? Many will opt for the latter way of talking. Now pluck the open string again and carefully bring a finger down lightly over the octave fret to create a high “harmonic”. Suddenly a new sound is heard: “purer” somehow and of course an octave higher. Some people insist that this is an entirely novel sound, while others will describe the experience by saying “the bottom fell out of the note”--leaving just the top. But then on a third open plucking one can hear, with surprising distinctness, the harmonic overtone that was isolated in the second plucking. The homogeneity and ineffability of the first experience is gone, replaced by a duality as “directly apprehensible” and clearly describable as that of any chord.

The difference in experience is striking, but the complexity apprehended on the third plucking was there all along (being responded to or discriminated). After all, it was by the complex pattern of overtones that you were able to recognize the sound as that of a guitar rather than a lute or harpsichord. In other words, although the subjective experience has changed dramatically, the pip hasn’t changed; you are still responding, as before, to a complex property so highly informative that it practically defies verbal description.

So what changes between people who can detect all the subparts of the sound of a single low guitar string, and people who can only hear the note as a single, intrinsic, undivided “guitarish” sound?

Does the guitar make different sounds based on how much you can detect? That seems unlikely. If one person can pick apart the sounds they hear much better, are they having additional sensations beyond the sound of the guitar itself, or is the guitar string just composed of these individual sensations like a checkerboard is composed of red and black squares, and the other person just can’t pick up on those internal differences? But “picking up on” the sound is itself what we’re trying to describe here: the direct, first-person experience of hearing. These experiences are specifically not supposed to be just the information we gather that we can write down; they’re supposed to be the first-person, in-the-moment undeniable experience. The fact that whether the two people experience the same or very different qualia is another hint that our sensory experiences are not simply displayed on a mental screen for us to watch like a movie.

Conclusion

We do not have any clear way of telling whether any of our qualia are actually “intrinsic” and just a bare experience that can’t be broken up into parts, or actually an assemblage of many individual experiences at once. When you first drank coffee, you had no way of telling that this taste you were experiencing was or was not an assemblage of many different other experiences at once. Just like the first-time coffee drinker, we have no way of knowing whether any one experience we have is actually intrinsic, or a combination of many different experiences at once. This again chips away at our image of ourselves as having direct, easy access to our subjective, first-person experience. People often talk as if qualia are these brute, intrinsic, foundational facts of our mental world, but it seems like we can barely tell if any one experience we have is actually composed of many others, or whether we ourselves are affecting the experience based on our judgment and reaction to it.

This all makes our experience seem very distinct from an inner movie theater. Unlike the movie theater:

• It is not obvious when something is a single basic thing (like a solid color on the screen) or whether it’s composed of many other things we’ll easily notice when we get more experience with it.

• Our judgment seems to directly affect our experience. We are not passively watching a screen of experiences and then judging and reacting to them after. The screen itself seems to radically change based on how we judge it.

2. Ineffable

Dennett argues that the ineffability of qualia is not due to qualia being some special, mystical property, but because they are “practically” ineffable. They are just highly specific, information-rich responses to the world that could in theory be described if we have enough experience with them. “Ineffability” is just the current horizon of our ability to analyze our own responses, not a fundamental property of the experience itself. This limit can be changed with training (like wine tasting or ear training).

The last example with coffee and the guitar string gets at this idea. Qualia might only be indescribable because we and other people are relatively untrained at detecting the communicable information they’re giving us. With training, people can learn to communicate a lot more relevant information about the sound of a guitar string or the taste of coffee. It might be that we’re mistaking our own current cognitive limitations for a fundamental property of the experience itself. Over time, experts get better and better at describing qualia.

On this point I mostly part ways with Dennett, where it does seem like some aspects of experience might permanently be walled off from public discussion. But this isn’t as relevant to the question of the Cartesian theater.

3. Private

Qualia are really really private

Qualia are fundamentally private. No one else can ever access what we experience.

Suppose we invent a machine that connects our brains, feeding your visual input directly into mine.

I report seeing everything you see. (This is the plot of the 1983 movie Brainstorms). I step back and gasp “Wow! Your qualia are completely different from mine! Your red is my blue! Everything smells and tastes different. This is crazy! We’re living in two different worlds. Qualia can be so different without anyone noticing.” We agree this is crazy. Suddenly, you scratch your head and say “Wait a minute, maybe we messed up somewhere. What if we plugged the wire that connected us in upside down on one end? Let’s just check to see if that changes anything.” We flip the plug upside down at one end and try it again. This time, we both agree that the other person’s experience looks totally normal.

What happened here? There are two possibilities for us: either we have different qualia, or we have the same qualia. There are also two possibilities for the cord connecting us: either it inverted the images the first time, or the second time.

Can we tell which is true based on the information we have?

In a connection where the qualia look weird to us, there are two possibilities. One is that we both have the same qualia as each other, and it’s the connection that’s inverting it. This would cause us both to say the other person’s qualia looked weird.

Another possibility is that the connection is correct, and we do have inverted qualia. This would yield the same reaction:

The situation where we confirm that what we see is normal also has two possibilities. It could be that the connection is reporting what our qualia actually look like to the other person.

But it could also be the case that the connection is inverting our different qualia, making them look normal when they’re actually different.

Let’s organize these situations by whether we have the same or different qualia, and whether the cord inverts the image.

Now, you want to test to see if the cord is right-side-up or not. You plug two people in. As they’re watching the other person’s vision, they say “that looks weird!” This is all the information you have to go on. You can listen to them both say “Wow… your blue is my yellow. Your red is my purple” for as long as you want, but this doesn’t really add any new information, because either way you face the same problem: there are two possible situations where the participants say the other’s qualia looks weird: the cord’s inverting their similar qualia, or their different qualia are transferred faithfully by the cord.

Given the information you have, you have no way to tell whether it’s the cord that’s wrong, or if the two people have different qualia. Your evidence perfectly fits both cases. If you flip the cord, and the participants say “now it looks normal!” you have no way of deciding between these two cases.

Again, you don’t know if the cord is actually showing that the two people have the same qualia, or if it’s now upside down. You have no way of telling if the cord is right-side up or upside down. Even a machine like this can’t ever actually give you direct access to the other person’s qualia, because neither has any way to externally test if the machine is getting it right.

So qualia remain fundamentally private. We have no final way of telling whether someone else’s qualia is the same as our own.

If qualia are this private, you do not have direct mental access to your own qualia

But if qualia are this completely and totally private, this also creates a problem for us. All our understanding of what qualia look “normal” come from our memories and intuitions about what the qualia looked like in the past. And we’re actually in the same relationship to our past self as the two people were in the last thought experiment.

We can imagine swapping the machine in the last example with a different machine. This time, only you get in. When you step out, all the colors in the world look inverted to you.

You’re stunned. The world is overwhelmingly alien. You ask “How did you invert my qualia?” The machine operator chuckles. “Oh, your qualia are actually the same as always. The machine uses a simple trick: We just edited your memories and crossed your memories with different colors. You now remember yellow as being blue, and blue as yellow, as an example.”

You’re very confused. “But the banana I see in front of me now is clearly blue!”

The operator says “Oh no, it looks exactly the same to you as it always did. It’s just that all your memories of yellow have been crossed with blue things. Like I bet that banana looks exactly like what you remember the sky looking like, right?”

You nod.

“Well if you go outside, you’ll see the sky’s never actually looked like that. It’s blue and has always been blue. We just swapped your memories, so now that color’s associated with what you think of as the color of bananas.”

You go outside, and the sky looks banana-yellow to you.

“No this is all clearly wrong. You did invert my qualia somehow.”

The operator scratches her head. “How would you tell the difference between an inversion of qualia and an inversion of your memories?”

“Well I just feel in my gut that this is wrong.”

“But that gut feeling is all your memories and built-up intuitions of the world.”

Hmm…

Later, a huge scientific discovery is made. Scientists have located qualia themselves, directly in the pineal gland of the brain. The operator uses this knowledge to build a second machine, this one really inverts your qualia and leaves your memories untouched. She invites you to try it out, but because the machines look identical, she realizes she mixed up which one is which. Is there any way at all for you to tell which machine is which based on the outcome of going through it? When you emerge, will you have any way of knowing at all whether it was your qualia or memories that were inverted?

In this example, memories are taking the place of the cord in the first example. When it comes to your qualia, your connection to your past self through memory is identical to your connection to the other person’s qualia through an unreliable cord. In both cases, you don’t actually have direct infallible access to the qualia.

This is a big problem, because all human thought happens in time, and the present is an infinitely small slice of time. If qualia are so private that they can only be accessed in the infinitely short present, we humans cannot actually access them directly, because we live and think in time.

Whenever your mind receives any information at all about qualia, you are reacting to a past experience, because of the fractions of a second it takes the signal from the qualia to be mentally processed. But already at this point, you’re thinking about something in the past, and you’re dependent on memory. Qualia’s privateness means memory does not give you that special access you’re supposed to have. Memory, like the cord, is fallible. You can be wrong about what you just saw. “What’s in front of your eyes” still needs fractions of a second to process. To be clear, this doesn’t mean you’re necessarily wrong about the qualia you’re experiencing, it just means you can be wrong about them. This strikes at the heart of the Cartesian intuition that we have direct, certain access to our own subjective experience.

If the qualia in the last experiment are so fundamentally private that no one else can ever access them, you yourself also have no fundamental cognitive access to your own qualia. They are as inaccessible to you as they would be to another person using the brainstorm machine to try to understand what it’s like to be you. Your brain can provide you with valid reports of what you’re experiencing, just like the other person can provide you with verbal reports about what they’re experiencing, but in both cases you don’t have that direct, magic, private access qualia are supposed to come with. In fact, here they’re behaving suspiciously similarly to a world where they don’t exist, and where your brain is only receiving information about the world that can be reported publicly, either to your future self or to another person.

Again, this seems like really bad news for the inner theater model of the brain. Not only is there no observer and the screen itself unclear and obscured by our own judgments on it, suddenly the screen’s in another room, and written reports are being sent in about what’s happening in the movie.

A relevant argument here is Derek Parfit’s long piece in Part 3 of Reasons and Persons on how the self does not persist over time, and your relationship to your past and future self is similar to your relationship to other people (summary here). This is an especially good but long read for people interested in more arguments against Cartesian egos. If qualia are so fundamentally private that no one else can access our qualia, but also we ourselves are different people moment to moment, the intersection of these views seems to imply that our own access to our past qualia (which is all qualia we can actually think about and respond to, because the present is an infinitely thin slice of time) is identical to our access to other people’s qualia: indirect and uncertain.

What to make of this? I think that this is a strong sign that qualia (whatever they are) are not fundamentally private, and might exist instead as some kind of (very complex) reportable information that can be communicated between people who can understand it. This makes much more sense to me than the idea that they are fundamentally private, because the implications of fundamental privacy make them behave completely differently than how I experience them. I want to define qualia as “the experiences that I have some direct access to.” It might be that those experiences are entirely just raw information in my brain assessing the situation. This information could in principle be communicated completely to another brain. I’d hope so, because I also think the relevant information can be communicated by memory across time to my present self. Again, this is a case where it seems like we have as much mental access to our own qualia as a computer would: an uncertain tether of memory or wires communicating information about what the world is like.

The more inaccessible you believe qualia are to machines, the more inaccessible you believe they are to your own mind as well.

4. Directly or immediately apprehensible in consciousness

If qualia are so fundamentally private, we’ve seen above that they by definition are not directly or immediately apprehensible in consciousness. We’ve also seen via the coffee and beer examples that people have a surprisingly hard time apprehending the qualia they are currently experiencing or have experienced in the past. We can’t even tell if the qualia we’re experiencing are basic and intrinsic, or composite and relational, as in the case of the guitar string. If qualia exist, we have a strange and complicated relationship with them. They are not directly apprehensible in consciousness in the way an unobscured movie screen is.

Conclusion

I think there are some clever arguments that even with these objections, qualia may still exist. But we’re way, way less sure of our own subjective experience than folk theories imply, and the nature of qualia seems more complex than simple, accessible, nonphysical pure experience.

So it seems like we don’t actually know our own internal experiences with certainty. Here I don’t mean that we might be mistaken about what objects and events we’re seeing, I mean that we can be mistaken about the nature of our current first-person subjective experience. Not only is there no one sitting in the Cartesian theater, and not only is the screen itself obscured, but there doesn’t seem to be a screen at all. This magical place, the center of the mind and thought, that so many people seem to take as so obviously given it’s crazy to doubt it, does not seem to actually exist.

If this is the case, first-person subjective experience becomes one of many possible sources of information and evidence. It’s useful, it’s just reduced to the same level as other ways we acquire information, like written descriptions, verbal accounts, and knowledge that other people we trust believe things (all of which computers can access). Most importantly, humans do not seem to have some magical, central location where experience happens that machines by definition will always lack.

The impossibility of folk introspection

A very common critique of large language models is that they can’t “introspect” in the way humans can. They just take in input, run it forward through the net, from its input variables to its final classification, and don’t have any way of circling back on anything that’s happened. They can’t go backwards. Therefore, unlike humans, they do not have introspection. They’re pattern recognition rather than true thought. Here, the information in the neural network passes from left to right through the weights and nodes. The only way this model could have true introspection is if it were somehow able to step outside of this process and observe what had already happened:

This critique always seemed implicitly Cartesian to me. Human brains are also only more or less complex information-processing units. They are composed of inputs, mental functions that perform computations, and outputs. Those outputs are themselves used later as new inputs.

Crucially, all information processing in the human brain can only happen in one direction in time: forward, because every mental subprocess in the brain is physical. When our minds take in input, they might combine that input with memories or inbuilt mental biases, run these through mental computations, and product an output.

This could get more and more complex, as different parts of the brain draw on results from the mental computations of other parts:

Where would introspection fit into this? Well, if introspection is another natural functional process, it has to exist in time, take in inputs, and give outputs. It seems like what we call “introspection” would be the brain calling prior inputs or results of past computations. Maybe it would look like this:

Crucially, what is NOT happening is that there is a magical outside observer looking down at the mental processes from outside. There is no Cartesian ego who can “just see directly” what’s actually happened in the mind.

“True introspection” where we step back and clearly look at our thought processes from the outside is not possible. Introspection is just another mental function that takes in inputs and gives outputs. We cannot actually see the wiring of our own minds, we can just react to the outputs our minds are giving us, in the same way we can observe someone else’s behavior over time and come to conclusions about what they are like internally. “Introspection” is actually just another informed guess about the world our brains make. Introspection is like observing a natural phenomenon and making predictions about what will happen next. It’s not like looking in a mirror and clearly seeing ourselves as we really are. Just like we can be mistaken about the real causes of other people’s behavior, we can also be mistaken about the causes of our own behavior, because the inputs our minds provide for introspection can be misleading. We can call on false or modified or selective memories with no way to tell that they’re not perfectly accurate. We can build up strong incorrect intuitions about what we’re like. There’s no special place our intuition function can turn to get the “real” story, it’s all fallible inputs just like the external world provides for anything else we’re trying to understand. I’d recommend this classic essay for a deeper dive on this.

It seems like when people say AI models do not have any way of doing introspection, they think that to do that, the model needs to have some outside Cartesian ego looking in:

But if introspection in humans is just another mental functionalist process, where inputs are received and outputs given, rather than a magical Cartesian ego looking down from above, then it seems like anything we call introspection in a process like this:

could in principal also happen in a process that looks like this:

The fact that LLMs “only pass information along one direction” on its own tells us nothing about whether they can introspect, for the same reason that the fact that human information processing can only happen in one direction in time tells us nothing about our own ability to introspect. The only way this would invalidate the idea that LLMs can do anything like introspection is if folk Cartesianism were true.

There are obviously a lot of other ways large language models differ from the structure of human brains. But any comment on those differences shouldn’t imply that the AI companies forgot to install a Cartesian ego watching over the whole thing.

A simple rule for what introspection can’t do

The individual thoughts and experiences we have access to during introspection are not the fundamental units of thought. The fundamental units of thought are the electrical signals sent and received by neurons. They combine together into thoughts in the way binary 1’s and 0’s can combine together to form a programming language. Most of our mental processes are very complex mental functions with neurons and electrical signals as the building blocks. Because introspection is another mental function, it does not necessary have access to every minute step of its own or other functions’ structure. Computer programs can summarize the outputs and steps of other programs, but they can’t “step outside the process” and observe the binary 1 and 0 signals that make them up. In the same way, human introspection can also draw information from other mental processes, but cannot “step outside” to see the mind laid bare before it, because the mind is not actually fundamentally composed of the type of thing introspection gives us access to.

Conclusion

Our assessment of our own internal experience and thought process is fallible in the same way our assessment of the external world is. There is no “solid ground of absolute certainty” we can find in our subjective experience. This makes sense of the mind is a pile of complex functions taking in information, processing it, and providing new inferences as output. There’s no deep metaphysical difference between the mind analyzing its own contents and the mind analyzing the world, for the same reason

Whether the information the mind receives is about what’s happening inside of it, or what’s happening externally to it, doesn’t have some deep metaphysical difference.

From the outside, our situation with introspection, and the vagueness and difficulty of directly apprehending qualia, give the impression to me of a species with very strong internal built-up narratives of how our minds work colliding with the simple fact that we’re machines and not angels. We can perform mental functions on the information we receive, but we cannot enter some kind of extra-physical movie theater of consciousness, where we easily and directly apprehend our deep consistent subjective worlds, and draw conclusions based on our assessments of those worlds. In this way, we behave similarly to very general computers sent off to survive in the world.

“3. The basis of all knowledge lies in our first-person subjective experience”

We’ve already seen that:

• There is no Cartesian ego observing and judging our experience.

• Our first-person subjective experience is surprisingly opaque. We can be mistaken about it. It’s still evidence, but we don’t actually have 100% confident privileged access to it.

• Similarly, our introspection is just another mental process that takes in fallible inputs and gives fallible outputs. The mind is not actually all easily available to us at once, like looking down at a complex domino chain falling. Anything we call “introspection” is just one part of that very complex chain itself.

Human minds take in input, run it through very complex mental functions, and give output, that they then use for other functions. In no part of this process does the mind “step above” the physical functions happening in the brain, and looks down at all the mind-stuff happening at once, and observes it like a map. Our felt sense that this happens, that we’re Cartesian observers with access to clear distinct instrinsic private mental experiences, is strong but mistaken. At best, the functions in our brains doing the introspection draw from lots of parts of the brain at once,

The folk theory of language acquisition

Many people believe we learn what words “really mean” by connecting them to specific experiences in our minds. They think we only understand “cat” because we’ve seen an actual cat and someone labeled that experience for us. Since your personal experience of seeing a cat lives privately in your mind, the “real meaning” of “cat” must be something private that only you and others who’ve had similar experiences can truly grasp.

But if word meanings are fundamentally private, language breaks down. If everyone has their own private version of what words mean, with no way to compare these meanings, how can we ever know if we’re talking about the same thing? For language to work, meanings must be publicly accessible and verifiable.

When I learn the word “cat,” its real meaning can’t depend on some private, indescribable feeling I experience, because you might experience something completely different. The meaning must be something we can communicate using other words and shared reference points. It makes much more sense to say the meaning of “cat” is something like “Small carnivorous mammals of the species Felis catus, typically weighing between 8-11 pounds, with retractable claws, whiskers, and forward-facing eyes, commonly kept as household pets and are known for hunting rodents and birds” than to say it’s something private and indescribable that people who haven’t seen one cannot possibly understand.

We only have access to public evidence about meaning. We can’t look inside each other’s minds. So any realistic theory of how we learn language must focus on what we can actually observe: how people use words in response to their environment.

We don’t learn language by linking words to mental images. Instead, we learn by linking words to publicly observable situations where they’re used. The philosopher Quine called this “stimulus meaning.” A child learns when to say “cat” by observing environmental cues and getting social feedback (nods, corrections) from the people around them.

A word’s meaning comes from its role in our entire language system, not from its connection to a single private idea. This explains how we use abstract words like “democracy” or “neutrino,” terms for which we have no clear mental picture or direct sensory experience. The old theory can’t account for these. We understand these terms not through mental images, but through their relationships to other concepts in our linguistic and social framework.

Since word meanings must depend entirely on things we can communicate about using language, and exist as essentially nodes in a web of interrelated words, they can’t depend on private, indescribable experiences. This means machines that process enough information can, in principle, access the meaning of words. They don’t need inner transcendent subjective experiences to understand language. Modern large language models are designed to situate words in broader webs of meaning and association. Once you exit the folk theory and learn about how LLMs process language via association, it becomes hard not to think that they’re successfully mimicking what humans do when the learn the meanings of words.

Can machines without qualia have knowledge?

This will be an argument that machines can in principle have knowledge. I’m not arguing that current AI systems have knowledge (though I would argue that chatbots do “know” the meaning of the words they use).

What is knowledge?

Knowledge is justified, true belief. It’s worth breaking this down a little:

• Justified: I wake up in my room with the window shades down. I can’t see if it’s raining outside. I decide to flip a coin. If it’s heads, I’ll believe that it’s raining. If tails, it’s not. I flip the coin and it lands on heads, so I believe that it’s raining. I go outside and see that it actually is raining. Did I “know” it was raining before I saw it? It seems like the answer is no. I got lucky and my silly guess happened to match reality, but to “know” it was raining I needed some kind of justification for my true belief.

• True: It doesn’t make sense to say I have “knowledge” that David Hume is currently President. We can only be said to have knowledge about true beliefs.

• Belief: An acceptance that a statement is true or that something exists.

Anything “completely private” cannot contribute to most knowledge.

Can machines hold beliefs?

Before we can ask if machines can have beliefs, we need to get clear on what beliefs actually are. The folk Cartesian story goes something like this:

A belief is a mental state that exists in your Cartesian theater. When you believe it’s raining outside, there’s some kind of propositional content—”it is raining”—that exists in your mind, in your subjective experience. You have direct access to this belief through introspection. You can “look inward” and see what you believe. The belief causes you to grab an umbrella, but the belief itself is fundamentally a feature of your private, subjective mental life.

But we’ve already seen problems with this picture:

• There is no Cartesian theater where beliefs are displayed for an inner observer

• We don’t have reliable introspective access to our own mental states

• Mental processes are just physical information-processing functions

So what’s left when we strip away the Cartesian story? What is a belief if it’s not a proposition floating in your mental theater?

In the physicalist-functionalist view, a belief is a certain kind of functional state. It’s a state that:

1. Takes in information from the world (perceptual inputs, testimony from others, memory, reasoning)

2. Combines with other beliefs and desires to guide decision-making

3. Produces behaviors that would make sense if the belief were true

4. Updates based on new evidence

5. Plays the right inferential role in reasoning (if you believe “All mammals are warm-blooded” and “whales are mammals,” you should infer “whales are warm-blooded”)

Notice that nothing in this definition requires a Cartesian theater, qualia, or privileged introspective access. A belief is defined by what it does, how it functions in the overall cognitive system, not by what it feels like “from the inside.”

You might object: “Fine, maybe machines could have states that function like beliefs. But real beliefs require understanding. When I believe it’s raining, I understand what rain is. A machine just manipulates symbols without understanding what they mean.”

But we’ve already undermined this picture in the folk theory of language acquisition above. We’ve seen that:

• Meaning isn’t grounded in private, ineffable qualia.

• Language acquisition works through publicly observable patterns of usage, not through attaching words to subjective experiences.

• Our own understanding is ultimately cashed out in how we use words, respond to contexts, and draw inferences, not in some private mental glow of comprehension.

If understanding is functional, if it’s about having the right patterns of response, the right inferential connections, the right sensitivity to context, then there’s no reason in principle that machines couldn’t achieve it.

Can machine beliefs be justified?

The folk Cartesian picture of justification goes something like this: Your beliefs are justified when they’re properly grounded in your direct, first-person experience. You know you’re seeing red because you have direct, certain access to your own qualia. You can introspect and observe the redness right there in your mental theater. This provides a firm foundation, a bedrock of certainty, upon which all other knowledge is built.

Descartes himself made this explicit. He doubted everything he possibly could. The external world, his own body, mathematics, even whether he was dreaming. But the one thing he couldn’t doubt was his own existence as a thinking thing: “I think, therefore I am.” His subjective experience provided absolute certainty that he existed and was having thoughts.

From this foundation of subjective certainty, Descartes tried to build back up to knowledge of the external world. The idea was that our private, first-person experience provides a special kind of justification that nothing else can match.

But we’ve already seen the problems with this picture:

• We don’t have direct, certain access to our own experiences

• Our introspection is fallible and often misleading

• The supposed bedrock of subjective certainty is actually shaky and unreliable

• Our experiences aren’t clear, simple, and directly apprehensible. They’re complex, constructed, and surprisingly opaque

• The Cartesian ego Descartes detected might not even exist

If the folk Cartesian story about justification is wrong, what’s left?

An alternative picture of justification is reliabilism: A belief is justified when it’s formed and maintained through reliable processes that tend to produce true beliefs. Justification isn’t about having some special subjective feeling of certainty or being able to introspectively verify your mental states. It’s about the process that produced the belief being a good one.

Consider how you actually form beliefs:

Visual belief: You look out the window and form the belief “It’s raining.”

• Your eyes receive light reflected off raindrops

• Your visual cortex processes this information

• Pattern recognition systems identify the characteristic appearance of rain

• Memory systems match this against stored representations of rain

• A belief state is formed: “It’s raining”

Is any of this process fundamentally dependent on having qualia or a Cartesian theater? It doesn’t seem like it. The process is:

1. Receive sensory input

2. Process that input through various computational mechanisms

3. Compare against stored patterns and previous experiences

4. Form a representation of the world state

5. Store this representation for future use

This is an information-processing story all the way down. And crucially, this process is reliable: it tends to produce true beliefs about whether it’s raining. That’s what makes the belief justified.

On the reliabilist account, justified beliefs don’t need to be certain. They just need to be formed by processes that usually get things right.

This actually matches our intuitions better than the Cartesian story. Consider:

You look out the window and see what appears to be rain. But it could be:

• A sprinkler system you forgot about

• A movie set creating artificial rain

• An elaborate hallucination (you’re on drugs, or having a stroke)

• A very realistic dream

• A brain-in-a-vat scenario where you’re being fed false sensory data

You can’t be certain it’s raining with the kind of absolute certainty Descartes wanted. But that’s fine. Your belief is still justified because it was formed by a reliable process. Most of the time when your visual system produces the representation “rain,” there really is rain. The process is reliable even if it’s not infallible.

What counts as a “reliable process”? Here are some features:

1. Correlation with truth: The process should produce true beliefs more often than false ones in the environments where it operates.

2. Appropriate sensitivity to evidence: The process should update beliefs when new evidence comes in, and it should be more confident when evidence is strong and less confident when evidence is weak.

3. Coherence: The process should produce beliefs that fit together consistently. If you believe “All mammals are warm-blooded” and “Whales are mammals,” your processes should lead you to believe “Whales are warm-blooded.”

4. Resistance to known error modes: The process should avoid systematic biases and known sources of error, or at least be able to correct for them.

5. Appropriate use of background knowledge: The process should integrate new information with relevant prior knowledge rather than treating each belief in isolation.

Could a machine implement processes with these features?

Let’s think about how a machine might form the belief “It’s raining”:

• Sensor input: A camera receives visual data, or a moisture sensor detects water.

• Pattern recognition: Neural networks or other algorithms process this data to identify rain patterns.

• Comparison with stored data: The system compares current inputs against training data or previous observations.

• Belief formation: The system forms an internal representation: “It’s raining.”

• Confidence assessment: Based on signal strength and pattern matching confidence, the system assigns a probability or confidence level.

Is this process reliable? Well, that depends on the specifics:

• How good are the sensors?

• How well-trained is the pattern recognition system?

• Does the system have enough relevant training data?

• Can it distinguish rain from sprinklers or other confounds?

• Does it update appropriately when conditions change?

These are all empirical questions. There’s no in-principle reason why a well-designed system couldn’t form beliefs through highly reliable processes. In fact, for some tasks, machine processes might be more reliable than human ones.

Under reliabilism, it looks like machines can in fact have justified, true beliefs. Therefore, it seems like machines can have knowledge.

The implication that humans might be incredibly complex biological machines rather than transcendent experiencing angels should hopefully be a little more understandable than when we started.

Conclusion

So I claim folk Cartesianism is not “obviously correct,” and a better story of the mind looks something like this:

• Humans do not have Cartesian egos, magical eyes floating outside of physical reality observing what happens in our minds like a movie.

• Human experience is deeply fallible as a source of information about our own minds, and even of the experience itself. We do not actually have a clear understanding of what’s happening in our own minds, or even a clear understanding of what we are currently experiencing at any given moment. “What it is like to be us” is not something our brains have special, ultimate access to. We have very strong evidence, but our minds are not displayed before us clearly like a movie.

• “True introspection” where we step back and look at our thought processes is not possible. Introspection is just another mental function that takes in inputs and gives outputs. We cannot actually see the wiring of our own minds. “Introspection” is actually just another informed guess about the world our brains make, this time based on the patterns we’ve observed in our own behavior. Introspection is like observing a natural phenomenon and making predictions about what will happen next. It’s not like looking in a mirror and clearly seeing ourselves as we really are.

• Human knowledge is based on publicly observable evidence and testable reasoning, not privileged access to private mental contents. Our beliefs are justified through the same kinds of information processing that machines can perform, drawing inferences from data, testing predictions, and refining models based on outcomes.

• The meaning of language is rooted in public use and social context, not private mental associations. We learn words through observable patterns of usage, not by linking them to ineffable subjective experiences. This means machines can acquire genuine understanding of language without needing a Cartesian theater.

• There is no fundamental barrier preventing machines from having beliefs, knowledge, or understanding. If our minds are physical information-processing systems (as physicalism suggests) and what matters is function rather than material (as functionalism suggests), then appropriately designed machines could replicate any cognitive capacity humans have.

I’ll close with a famous line from Derek Parfit on the reductionist view of personal identity, where he describes the experience of letting go of his sense of having a Cartesian ego and internal movie theater.

On the Reductionist view each person’s existence just involves the exercise of a brain and body, the doing of certain deeds, the thinking of certain thoughts, the occurrence of certain experiences, and so on…

Is the truth depressing? Some may find it so. But I find it liberating, and consoling. When I believed that my existence was such a further fact, I seemed imprisoned in myself. My life seemed like a glass tunnel, through which I was moving faster every year, and at the end of which there was darkness. When I changed my view, the walls of my glass tunnel disappeared. I now live in the open air. There is still a difference between my life and the lives of other people. But the difference is less. Other people are closer. I am less concerned about the rest of my own life, and more concerned about the lives of others.

Subscribe now

“Let me see if I understand your thesis. You think we shouldn’t anthropomorphize people?” -Sidney Morgenbesser to BF Skinner

The 2020s have made my philosophy degree feel more worthwhile. Not only has social media made a ton of new people interested in philosophy, but the real world has also made many of the main questions in philosophy more relevant. AI especially has brought many philosophy topics into the mainstream, but it’s also polarized a lot of the debates. Back in 2011, I could say things like “The human mind is composed of natural processes, like everything else in the world, and therefore it could probably eventually be replicated by a machine, because in some sense it itself is a machine” and people would have long drawn-out conversations with me about whether that’s true or what that implies about language or experience or society. Now, when I say the same thing, it’s much more likely that I’ll get accused of “just buying into the recent AI hype” or that I’ve been tricked by secularism or capitalism or worse to deny the fundamental transcendent character of human minds. This issue used to be a pretty marginal question, but now it’s entered mainstream partisan debate. Sides have been taken.

What’s odd is that a lot of critics of the “minds are machines” view used to implicitly defend it. Back in the 2010s, there was a general worry about anthropocentrism. The belief that humans were in some way unique or separate from the natural world was regularly sneered at as “naive humanism” at best and a method of justifying exploitation of the natural world at worst. There was also a lot of talk about Cartesianism (the philosophy of René Descartes) as the origin of many of society’s problems. Specifically, mind-body dualism was accused of denigrating embodiment and elevating technocratic authoritarian reason. I always thought both criticisms went too far, but analytic philosophy had turned me into an anti-Cartesian naturalist, so I was able to nod along and contribute to these conversations where I could. Since then, attitudes toward Cartesianism have flipped. Many of the same people are now saying that AI cannot, by definition, ever have knowledge, because it does not have subjective first-person experience, or that it’s lacking some transcendent non-physical characteristic the human mind has; both implicitly Cartesian takes. Accusations of anthropocentrism have also been replaced with accusations of anthropomorphizing. A common line is that “everyone who actually knows how this technology works knows that it’s just linear algebra.1 The only people who believe it’s thinking are dummies who anthropomorphize it because it seems human.” I long for the 2010s when it was more common to hear people talk about a lot of different systems exhibiting some basic qualities of mind. I think back then people would be less likely to assume linear algebra couldn’t mimic some basic mental processes.

There’s a cluster of beliefs I associate with “folk Cartesianism” that even critics of Descartes often fall into. It’s incredibly common, to the point that people talk as if its implications are obvious without ever having directly thought about them before. I think this cluster of beliefs falls apart when you poke at it, and the fact that it’s implicit in so many debates about AI shows that it’s not being poked at enough. This post will be me poking at folk Cartesianism, and arguing for “anti-Cartesian naturalism” which is a widely-shared view among analytic philosophers of mind. Nothing I write here will be new or surprising to people knowledgeable about analytic philosophy. But I’ve been feeling the gulf between academic philosophy and mainstream discourse and want to contribute to building a bridge. This will be a long collection of the most popular and effective arguments for why folk Cartesianism is wrong, and how I think it’s harming the AI debate.

I have three goals here:

• You can identify what folk Cartesianism is, and when people are thinking like folk Cartesians.

• You come away no longer viewing folk Cartesianism as the obvious, common-sense view of the mind. You have a better understanding of the anti-Cartesian naturalist view, and philosophy of mind in general.

• You are more open to the idea that machines can replicate what happens in human minds. That minds are not magical, inexplicable substances separate from nature.

I’ve written this partly as an instruction manual to my past self I wish I had when I was younger, as my personal crash course in philosophy of mind 101 as it relates to AI.

Some of these points will be taken from my other posts on the same ideas. The “Naturalism” section will be repetitive for people who have read those already.

Subscribe now

Contents

Part 1: Defining the problem

• Definitions

  • Folk Cartesianism

    I define folk Cartesianism as consisting of three common intuitions:

    1. There is a unified self that floats above mental processes and observes them

    2. We know our own internal experience with complete certainty

    3. The basis of all knowledge lies in our first-person subjective experience

    • The Cartesian view of the mind

  • Naturalism

    • Definition

    • Physicalism

    • Functionalism

    • Evolution

Part 2: Problems for Cartesianism

• “There is a unified self that floats above mental processes and observes them”

• “We know our own internal experience with complete certainty”

• “The basis of all knowledge lies in our first-person subjective experience”

Definitions

Folk Cartesianism

René Descartes inaugurated modern philosophy by attempting to establish a way to reach ultimate truth completely dependent on reason alone (rather than religious revelation). To do this, he looked for a solid, undeniable foundational fact he could start at. Everything that followed would be equally certain. He famously attempted to doubt all his beliefs, and found that the one belief he couldn’t doubt is that he himself existed. “I think, therefore I am.” From here he reconstructs all his other beliefs by building an intellectual ladder, with his belief in himself as the foundation. The next step is a pretty weird proof of God, followed by concluding that God wouldn’t deceive us about the external world.

Descartes’ explicitly and implicitly put forward many more specific claims about the mind, and his overall account matches how many everyday people imagine their minds working. “Folk Cartesians” mostly haven’t read Descartes, but have been influenced by how he and the broader culture talk about mind. So when I talk about “Cartesianism” here, I mean a cluster of concepts that are popular in everyday folk theories of the mind, not necessarily what literal Cartesian philosophers believe.

There are three key features of Cartesianism that I want to deny, and that are relevant for current conversations about AI.

1. There is a unified self that floats above mental processes and observes them

A “Cartesian ego” is a featureless observer hiding in the mind. This is the “I” in Descartes’ “I think therefore I am.” Cartesians imagine the mind as something like a movie on a screen, with the ego the silent observer. Critics call this the “Cartesian theater” model of the mind.

2. We know our own internal experience with complete certainty

Cartesians would say that we ourselves are egos. We observe our first-person experiences and thoughts. We have direct, clear access to them, in the way someone at a movie theater has access to the picture on the screen. The Cartesian ego also has direct, unmediated access to thought processes. We can turn our inner eye toward our thoughts in the way we would a piece of paper that lays out a step-by-step argument. We can observe the mental moves we’ve made and introspect on these thought patterns by just observing them directly. When we introspect, what’s happening is that we’re stepping back and observing the overall shape of our thought processes as they happen, like watching a line of dominos fall while standing above them.

3. The basis of all knowledge lies in our first-person subjective experience

Anything we could call real knowledge of the world has to come from the Cartesian ego observing and experiencing the world, or introspecting on its thought processes. A machine cannot have knowledge, for the same reason a line of dominos cannot have knowledge: there’s no Cartesian ego behind the scenes experiencing and judging the world. Thus, true artificial intelligence is impossible by definition, and many go farther and say AI is useless at what it’s trying to do. It can only ever imitate and copy knowledge. It cannot have knowledge of its own. It will always and only be glorified auto-complete. Even a machine that can handle and process large quantities of information cannot have knowledge, because the mind does more than information processing.

Many people apply this to our understanding of language as well. We only access the meaning of words via our first person subjective experience. I only know what “blue” or “tree” means because I have had subjective, first-person experiences of blue things and trees, and have learned to associate the words with those experiences. More complex concepts like “the national debt” are made up of simple concepts I’ve experienced like “money.” Any entity without a Cartesian ego cannot actually understand the meaning of words, because it cannot have these foundational experiences where it learns what words really mean.

Together, these three beliefs guide how many people think about their own minds, and how they think about AI. I will argue that they are incorrect.

The Cartesian view of the mind

The three of these paint an overall picture of the mind: a rich, internal theater where our egos live and experience things. Most of the important stuff that we consider “mind” happens here. Mathematical functions performed on our experiences and ideas happen a little behind the scenes, and we can perform those functions in our little theater as well (like when we hold numbers in our head while adding them together). Computers can replicate these mathematical functions, but they cannot replicate the internal theater. My goal for this post is to build some intuitions for the idea that this theater doesn’t exist. That we are not egos perceiving clear, unambiguous experiences, and that the mind is actually functions all the way down, where “function” is defined as any process that takes in inputs, performs computations on them, and gives an output.

Naturalism

Definition

I’m arguing for “anti-Cartesian naturalism.” Philosophical naturalism is a loose concept. It basically means “the world follows the laws of science, there is nothing magical happening.” Most analytic philosophers are naturalists, they just have strong disagreements about what counts as part of the natural world. There are three concepts associated with naturalism I adhere to in philosophy of mind:

• Physicalism

• Functionalism

• Evolution

I’ll summarize and give some basic arguments for each.

Physicalism

Physicalism is the view that the only things that have causal effect on physical objects are other things described or used by the laws of physics. Physicalism could be said to mean “Ultimately, what exists is the universal wave function.”

For something nonphysical to have causal effect on physical entities would violate our current best theories of physics. Anything new that has causal effects is likely to be an entity similar to what physics is already describing. Physics has been remarkably successful at describing the world, and physicists operate under the assumption that anything influence physical objects will itself also be governed by physical law.

These are some qualities that all physical entities seem to share, and that differentiates them from supernatural entities:

1. Spatiotemporal location: Physical things exist in space and time. They have locations, exist for specific periods of time, and can be detected at specific coordinates. Supernatural entities are often conceived as existing “outside” space and time.

2. Causal structure and lawfulness: Physical things behave according to regular, mathematical laws that describe their interactions. These laws are:

   1. Discoverable through observation and experiment

   2. Consistent and predictable

   3. Expressible mathematically

   Supernatural things are often defined by their ability to violate or transcend these laws.

3. Detectability/measurability: Physical things can be measured, either directly or through their causal effects. They leave empirical traces. Dark matter counts as physical even though we don’t know what it is, because we detect its gravitational effects.

4. Public accessibility: Physical phenomena are intersubjectively observable. Multiple people can measure and verify them using instruments. They’re not private or accessible only to special individuals.

There are entities in the universe we don’t understand, like dark matter and energy, but we should still expect those entities to conform to these 4 patterns when we eventually learn more about them. While physicalists believe that many more things “exist” besides fundamental particles and fields, those things are ultimately just names for those fundamental particles and fields interacting in some way. Even something as abstract as “the national debt” can in some sense be reduced to the incredibly complex behavior of the physical particles that make up people on Earth.

Functionalism

From the Stanford Encyclopedia of Philosophy:

Functionalism in the philosophy of mind is the doctrine that what makes something a mental state of a particular type does not depend on its internal constitution, but rather on the way it functions, or the role it plays, in the system of which it is a part

Functionalism’s main claim is that mental states are defined by what they do, not what they’re made of. Among professional philosophers, functionalism is the most popular account of mental states.

A mathematical function takes an input and produces an output. Functionalists imagine mental states as being functions that take specific inputs and give outputs. A good example of something else like this is monetary value. “Twenty five cents’ worth of value” can exist in a quarter or a digital bank record of $0.25. What makes it the same monetary value is the function it plays in our broader system of exchange. When you transfer it to someone else, you can expect their behavior to be the same regardless of whether it’s delivered as a physical object or a digital exchange.

Similarly, the state of being in physical pain can happen in different animals with different physical brain structures, or even brains made of different physical substances, provided that their mental architecture reacts similarly (gives different outputs) to the same inputs. Just like the monetary value of twenty five cents can be instantiated in different physical systems (a metal coin or a computer), mental states as functions can also be instantiated in different physical objects.

If a system is able to reproduce all the same inputs and outputs as mental states as they occur in human minds, and is just made of different physical material, we should probably assume that it also has the same mental states as the human brain.

I don’t think current AI as it exists is conscious, but it is possible in principle for a machine made of silicon to be conscious like humans if it can reproduce the same inputs and outputs of our relevant mental states. It might be that at a certain level of complexity AI could also have mental states vastly alien to what humans can experience, but that we would still consider conscious experience.

An argument for functionalism, presuming physicalism

Suppose that we were able to replace exactly one neuron in my brain with a silicon chip that could send and receive exactly the same electrical impulses at exactly the same times as the neuron. Maybe there’s something about the substrate of silicon that prevents it from replacing all the functionality of a neuron, but let’s presume for now that it can send and receive all information the neuron receives. What would happen?

Importantly, if physicalism is true, nothing about my behavior would change. There wouldn’t be some immaterial soul or something similar swooping in to say “Hey, Andy’s brain has something strange in it. Something’s off!” The same electrical impulses would happen in my brain either way and cause me to behave in the same way. The same inputs would lead to the same outputs. If I looked at a tree and said, “I see a tree” before replacing a neuron with a chip, I would say the same afterward.

So if physicalism is true, my behavior would stay the same after my neuron is replaced with a chip. What would happen to my conscious experience of the world? Let’s imagine I’m staring at a tree (I’m using an example of my conscious experience of my field of vision because it’s easier to make an image). It seems like there are maybe 4 options, shown below:

• Option 1: No change - My conscious experience of the world doesn’t change.

• Option 2: A little missing - Maybe some small specific part of my field of vision or something similar goes blank.

• Option 3: Less consciousness - There’s less conscious experience in general. Maybe my experiences are more blurry or dim, like I’m in a dark room.

• Option 4: No consciousness - I no longer have conscious experiences.

We’ve established that my behavior wouldn’t change. Physicalism implies that the same inputs would lead to exactly the same outputs. This would make option 2, 3, and 4 very strange, because I wouldn’t react to my conscious experience changing, because my outward behavior would be exactly identical to the before case. If I looked out at a tree, and my conscious experience included a small piece of the picture missing, I would notice that and comment on it: “What’s going on? There’s a black dot in my field of vision.” If there were less conscious experience in general, I’d probably say “My vision is suddenly very blurry.” If there were no conscious experience, I’d say “Aaaah! I can’t see!” Because my behavior doesn’t change, I don’t say any of these things, I say “Look! A tree!” This would be very strange behavior if 2, 3, or 4 were true.

I trust my current self’s reports about his conscious experiences. My speech about my conscious experience matches the experiences I’m having. I should probably trust silicon neuron brain Andy’s reports as well.

Maybe it’s possible that my conscious experience would be disconnected somehow from my physical behavior, where I’d be helplessly losing consciousness as my physical body behaved normally as if it were a robot. Unconscious silicon brain Andy would take over as normal brain Andy helplessly lost consciousness and died. This would be very strange, because most of my day to day behavior is responding to my conscious experience. If you somehow removed my conscious experience’s ability to affect my physical behavior, I’d behave in very physically different ways, because I’d be missing a lot of important information. I wouldn’t react to heat or cold or anything in my field of vision. I might behave as if I were dead. It seems like it’s very likely that my conscious experience wouldn’t be affected by replacing 1 neuron with a silicon chip.

We could then go ahead and replace a second neuron with a silicon chip. For the same reason, my conscious experience wouldn’t change. We could build special chemical receptors and signals to replace other parts of my brain. As long as they had the same inputs and outputs on my behavior, I wouldn’t expect my conscious experience to change. At the end, I’d have the same mental states as before, but my brain would now be made mostly of silicon and other materials. This thought experiment is evidence that functionalism is correct: what makes something a mental state depends on its inputs and outputs, not on the specific physical material the system creating that mental state is made of.

So we’ve established that if physicalism is true, it seems likely whether my mind is in a certain conscious state is that the same inputs lead to the same outputs, not the internal constitution of what’s creating those inputs and outputs (a neuron vs. a silicon chip). Physicalism seems to imply functionalism.

Two other key ideas that together seem to imply functionalism:

• The Neuron Doctrine: The idea that the brain is made of discrete cells (neurons) that communicate through connections. This supports functionalism because it suggests the brain works through information processing between units, not through some special physical substance. This is a central tenet of modern neuroscience.

• The Church-Turing thesis: Any function that can be computed by any possible mechanical process can also be computed by a Turing machine. In other words, there’s a fundamental equivalence to all forms of computation. They’re all doing the “same thing” at a deep level.

If the mind is instantiated by a system of neurons taking inputs and sending outputs to each other (performing specific functions), and that this exact same type of thing can be instantiated in many different mediums, this seems to imply that everything we think of as mental can in principle happen in other systems that can perform computations.

It’s important to note that many non-physicalists are also functionalists, David Chalmers being the most famous example.

Evolution

If naturalism is true, the mind is a product of evolution. Like all other parts of all animals, the mind was subject to evolutionary pressures and natural selection. We should not expect that the mind came pre-made to access abstract truths about the world. Instead, for most of life’s history, minds mainly operated as machines to predict very local environments and specific situations, help the animal survive and reproduce, and (in more advanced brains) achieve and compete for social status (social status here being very broad, and including things like friendship and belonging). Given our evolutionary history, the real question is “Why do people believe true things?”

Dan Dennett had a great line about how “conscious human minds are more-or-less serial virtual machines implemented inefficiently on the parallel hardware that evolution has provided for us.” To say this in everyday language, our ability to do abstract reasoning, to follow a logical argument from premise to conclusion, to work through a math problem step-by-step, or to carefully consider the pros and cons of a decision, runs on top of brain hardware that was designed for the big pile of parallel unrelated background mental tasks we’d need to deal with in the evolutionary environment.

Our brains evolved to handle lots of different things at once. We need to monitor our surroundings for potential threats, keep balance, regulate our body temperature, process other people’s faces and voices, and manage our movement. This massively parallel processing is what kept our ancestors alive. In comparison, conscious, deliberate reasoning was rarely useful in the ancestral environment.

This helps explain why certain mental tasks seem to require so much effort. When you’re trying to hold multiple abstract ideas in your head at once, or forcing yourself to focus on something boring, you’re essentially running software that’s fighting against the grain of the hardware. Recognizing a friend’s face in a crowd or catching a ball requires vastly more computation, but feels effortless, because that’s what the parallel hardware excels at.

We shouldn’t expect our conscious minds to be perfectly rational machines, or to have some kind of magical access to higher realms of truth than what is available to other complex physical systems. We can obviously access truths that other systems can’t, but our access to these truths demands physical and evolutionary explanation.

Subscribe now

Subscribe now

This is my favorite Sora video, and AI video in general:

Watch that again, focusing on how the way the horse’s hooves move is so coherent and makes sense given what it’s doing.

This is obviously slop. It’s a meaningless short video that won’t benefit me at all beyond activating my dopamine for a moment. But it’s also a sign of pretty remarkable ability.

Sora wasn’t told the laws of physics, or how horses move, or what skateboards are. It was able to infer all this from scratch from intaking gigantic amounts of video data. By getting more and more data, and having more and more internal “knobs” it could adjust to respond to that data, it was eventually able to infer enough useful heuristics that it can create a convincing video of a horse riding skateboards. This is an especially great new video explanation of how this works:

There are a few things happening in AI right now that make objective assessments of its capability difficult. Among other things:

• People are constantly overhyping what it can currently do, leading to general skepticism when it doesn’t match up.

• The data center buildout has been massive, and headlines about the cost and impacts have led many to believe that AI needs to be a world-historic technology right this second to justify it.

It’s very easy to react to any new improvement in AI by just immediately identifying all the ways it doesn’t measure up to the buildout or the hype. There are also a lot of in-the-moment status games to win. But try to think back to yourself 5 years ago. Would you have been able to predict that AI video would be able to do this in 2025? Here was the best state of the art research on AI video at the time:

Now we can make videos like this:

Sora and chatbots both depend on the transformer architecture, the pattern-recognition and heuristics-generating tool that’s been the backbone of most AI progress over the last decade. Transformers take in huge amounts of training data, and adjust huge amounts of internal parameters to pick up on more and more subtle heuristics and patterns. This ability to detect such complex and nuanced patterns is on display in the horse video. There is no way you could personally code clearly-written rules into a machine to make it so generally knowledgeable about the subtleties of video and motion that you could just type “Make a horse jump a ramp” or “Make Spongebob get a ticket” and expect it to give a useful output. This is why deep learning is so useful. It’s a way to get a machine to build its own soup of internal heuristics for how to handle complex situations. This soup of heuristics both succeeds and fails in surprising ways. We don’t design the heuristics themselves, and don’t even know what they are or how they work, in the same way we don’t know how a lot of the heuristics in our own brains work. We can now make machines that can pick up on such subtle heuristics that they can infer how a horse would look skateboarding over a ramp to the point that their output looks photorealistic.

It seems obvious that if deep learning can pick up on the patterns required to make the horse video, it can probably be used in a lot of other more important places in society. If it can independently figure out some rough approximation of the laws of physics to the point that you can be tricked into thinking videos of complex motion are real, surely it can help with weather forecasting, or have a huge number of specific applications for simple problems in climate, or might soon pose some new emergent risks? Sora 2 makes it especially hard for me to understand how anyone can say that deep learning models are “just glorified auto-complete.”1 They seem to be able to develop some pretty useful internal heuristics that correlate pretty strongly with what’s actually up in the world. If “glorified autocomplete” systems can produce photorealistic videos of anyone in any describable situation, the term stops being useful as a way of predicting what models can and can’t do.

ChatGPT has only been with us for 3 years. The time from the release of Windows 95 to Windows 98. The jump in AI capabilities has seemed much more significant to me over these 3 years than the jump in usefulness from Windows 95 to 98. An AI having the ability to beat out TikTok and Instagram as the most-downloaded short form video app (even for a moment) would’ve been unthinkable… a few weeks ago? Now I’m seeing Sora videos flood Instagram Reels. AI beating the slop machines at their own game, even temporarily, just seems like another win for people in the “AI is shockingly capable, and the capabilities are improving fast, so this really really doesn’t seem like a fad” camp. I’m worried that a lot of people are getting distracted by the stupidity of the outputs to notice how strange of a world we’re living in now.

Subscribe now

First, 2 definitions:

• Heuristics vs rules: Rules are absolute, fixed principles that must be followed to guarantee a specific outcome, while heuristics are "rules of thumb" flexible, practical shortcuts or educated guesses used for efficient problem-solving and decision-making, though they are not guaranteed to be optimal, perfect, or even correct.

• Deep learning: The specific type of AI that’s become dominant over the past decade, where instead of humans writing out explicit rules, we build large neural networks that learn their own internal rules or heuristics from data. These models stack many layers of simple mathematical functions, allowing them to automatically extract patterns from raw inputs (like pixels, audio, or text) and make increasingly complex decisions. Basically all debates about “AI” right now are debates about deep learning specifically.

Deep learning is a way to get a machine to build its own soup of internal heuristics for how to handle complex situations. This soup of heuristics both succeeds and fails in surprising ways. We don’t design the heuristics themselves, and don’t even know what they are or how they work, in the same way we don’t know how a lot of the heuristics in our own brains work. There’s a whole field of AI interpretability attempting to understand AI models’ internal heuristics. It’s hard!

Could you write a clear step-by-step algorithm to help a machine determine when a song feels nostalgic and when it doesn’t? Can you identify when a song feels nostalgic when you hear it, even if you don’t have the emotional reaction of feeling the nostalgia itself in the moment? If you can identify nostalgic songs, there are heuristics in your brain that you can’t articulate, and that maybe aren’t really articulable using our current language or methods of consciously thinking about the world.

Having some way to get a machine to build up similar inarticulable intuitions that lead to good results like this seems useful. The main benefit of deep learning is that before it, basically everything we used machines for had to be articulable in clear language and logic. If there were situations that we could build up tacit knowledge for how to handle, but couldn’t possibly write that knowledge down as clear instructions, machines couldn’t join us in navigating them. Now (with many bumps and failure points) they can.

Subscribe now

A funny example of this is chicken sexing. There are no clear written rules for identifying whether a newly hatched chick is male or female. The average person looking at one would just take a wild guess. Something odd happens where, if humans train for months, they can achieve 97% accuracy in looking at a newly hatched chick and identifying whether it’s male or female, but no one who can do this can articulate how, so they can’t write an instruction manual new people can use to skip those months of training. Our brains develop some complex internal heuristics for what male vs. female newly hatched chicks look like, but these heuristics aren’t actually available to our conscious minds. Until deep learning, we couldn’t build machines to do this task for us, because we couldn’t articulate what we’re doing when writing the machine’s code or working on its physical design.

If you train a deep learning model on enough images of chickens, it can also pick up on whatever opaque heuristics the humans are using. It also can’t articulate what those heuristics are, but whatever internal rules it developed are good enough that the machine can distinguish male from female chickens at 98.5% accuracy, beating out humans.

To get a sense of why and how deep learning models can develop these inarticulable heuristics, I really cannot emphasize enough that you should watch the 3Blue1Brown series on neural nets. It’s the best popularization of how deep learning works anywhere.

If a tool can develop heuristics that are inarticulable to itself and us, this implies 2 things:

• The number of possible use cases for it is gigantic. Computers took off so fast because they were so uniquely useful at solving problems that had clear logical steps we could articulate to them. This type of problem is very common. Deep learning models can help us with the other type of problem humans work on: places where we can’t articulate perfect rules to follow, but we know solutions when we see them, and experienced humans can develop tacit knowledge about a problem and solve it using intuition. There are a lot of examples of this type of problem as well.

• There will be a lot of weird ways these machines will fail, and they can’t be completely relied on in the way we could rely on computer programs with clear step-by-step instructions we could refine and test. This is the first time we’ve been consistently using machines with complex webs of rules and heuristics that we ourselves didn’t program into them. Instead of saying ChatGPT was programmed, it probably makes more sense to say something like it was “grown” in the lab during training. The potential for failures here is high (hence why AI models hallucinate and lie more consistently than normal computer programs), but the fact that they can develop their own heuristics at all makes them valuable even with a higher chance of messing up.

This is why many uses of deep learning (including ChatGPT) seem valuable to me, even though they can make a lot of mistakes.

Subscribe now

I’m posting this for two reasons:

• I think a lot of people still don’t know how good AI art’s gotten. I’m surprised that I haven’t seen more people playing around with it, or making the type of images I personally like. If we have the same aesthetic taste, you’ve also probably thought a lot of popular AI art looks kind of ugly. If we enjoy the same vibes, I’m hoping this post will convince you that AI tools can help you make a lot of images you find interesting and fun.

• I get a lot of good feedback when I share AI images I’ve made on Twitter, so I want a single place to collect the ones I like the most. I’ll use this post as a place to dump basically every AI image I’ve made that I really like.

Obviously it’s a philosophical question whether I “made” these, or whether they count as art. If you prefer, we can say I “found” these images in MidJourney’s vector space. I’m excited that I found them in the same way I’d be excited about finding a neat rock. I don’t think much of my own creativity went into this, and I know it’s different from creating art myself. Curating AI art feels kind of like curating a Pinterest board. I think that can be a neat way to spend time to develop and show off your own taste even if you’re not putting any creative effort into the images themselves.

When I have time, I’ll go back through these and add the prompts I used for each as alt-text. If you’d like the prompts for any of these in the meantime please let me know!

There are 152 images here. According to the MIT Tech Review, this means that altogether I used about 185 Wh of electricity to make these. That’s about the same energy you use running a normal kitchen microwave for 12 minutes, or playing a gaming console like a PS5 for about an hour. The average American household uses this much electricity every 8 minutes.

Subscribe now

Solarpunk landscape architect sketches

Some variation of “Solarpunk landscape architect sketch, watercolor” plus a lot of specific descriptions works well as a prompt here.

MidJourney now lets you animate every image:

Bit art of utopian architecture and communities

General weird architecture

Worlds made of weird textures

Album covers

Spheres

Brutalist monuments in forests

Animals (mostly pigs)

Cities and towns

Cafes & diners

Photorealistic/can you believe this is AI?

Art Nouveau

Old photographs

Maps & aerials

Surreal

Misc

Subscribe now

Me: I think a lot of people are getting a lot of misinformation from TikTok, and I think it’d be better if TikTok didn’t exist, but I’d consider anyone who said that TikTok is completely useless to its users to be pretty goofy. I feel the same about chatbots.

User: How can you both think TikTok shouldn't exist and think that it's useful to its users, without using a pretty unique definition of useful?

I was surprised by this reply. It showed up in a lot of other places too. To me it seems obvious that something can be useful in specific situations, but still be bad overall. Here are some examples:

• Nuclear weapons: I would prefer a world where nuclear weapons didn’t exist, but nuclear weapons are extremely useful to the specific countries that possess them. It seems ridiculous to call nuclear weapons “useless.”

• Guns: I’d support strict gun control if I thought it worked, and I think guns often harm their owners, but there are a lot of specific cases where guns provide a lot of value to their owners. Guns aren’t “useless.”

• Alcohol: I think it’d be good for everyone to boycott alcohol, because the effects its having on the people addicted to it are so bad, but alcohol obviously also adds a lot of value to specific people’s lives. It’s not “useless.”

• TikTok: TikTok seems pretty addictive, distracting, and contains a lot of misinformation and weird ideological rabbit holes. It’s also fun to use and can teach you a lot of specific social cues and context. I think it’s overall bad for society, but it’s definitely not “useless” to the people on it.

When I say that it’s not debatable anymore that chatbots are sometimes useful, a lot of people read me as saying that it’s not debatable that chatbots are good overall, and that all critics of chatbots are crazy. I don’t believe this at all.

Subscribe now

I think it’s totally rational to think that chatbots are bad overall. I could totally buy this criticism:

• Chatbots hallucinate a lot and give lots of incorrect information.

• The everyday person thinks they can rely on chatbots for correct information. They’re receiving more incorrect information than they would using traditional sources. This nets out to being bad overall.

• In addition, chatbots are giving people too many opportunities to cut corners on tasks they need to think about themselves. They’re very bad overall.

Any critic saying this could also agree that there are a lot of specific places chatbots are useful to people. For example, the only times I ask chatbots for contentious important information, I also ask them to provide sources, and I always check the sources. Most of the time, I’m asking them for much simpler lower stakes things, like sorting lists or doing simple back of the envelope calculations. Here’s a specific example where I get a lot of value out of using a chatbot. Any critic should be able to agree that those are examples of a chatbot being useful.

A lot of people do still talk as if chatbots cannot ever be useful for anything, period. People will question why I’d ever poke around on ChatGPT at all, or consider using it for literally anything. I think those people are obviously just wrong and silly. They’re making a fundamentally different claim than the people saying that chatbots are bad overall.

It’s no longer debatable that chatbots are useful in specific instances. It’s obviously very debatable whether chatbots and AI more broadly are good for the world overall. Clarifying that these are two different claims seems important if we want conversations to go well.

This is wrong. Most people aren’t super knowledgeable about a lot of specific facts about the world (only half of Americans can name the 3 branches of government) but they’re intelligent when it comes to their own lives and the areas they work and spend time in. We should expect the average person to struggle with factual questions about abstract ideas and far-off events, but not so much about what’s right in front of them day to day.

If a claim about how society works implies that most people are incredibly stupid, much more stupid than anyone I encounter in my day to day life, I dismiss it. This simple test kills a lot of big claims about how the world works. I’ve been applying it in a lot of AI conversations recently. I’ve written about this a bit before but want to go into more detail.

Here’s a common claim that I think fails my test: “The reason Americans are so unhealthy is that doctors don’t tell people about healthy diets.”

I think most people know what’s considered healthy food. They maybe wouldn’t be able to perfectly break down ideal ratios of macronutrients, but they have a rough idea. The average person whose bad diet is making them unhealthy would probably be able to point to the bad diet as part of the problem. If I walked up to the average person and asked them to make an ideal meal plan for themselves to be maximally healthy, I think most people would do a decent job.

Stefan Schubert makes a similar observation about what he calls sleepwalk bias:

When we predict the future, we often seem to underestimate the degree to which people will act to avoid adverse outcomes. Examples include Marx's prediction that the ruling classes would fail to act to avert a bloody revolution, predictions of environmental disasters and resource constraints, y2K, etc. In most or all of these cases, there could have been a catastrophe, if people had not acted with determination and ingenuity to prevent it. But when pressed, people often do that, and it seems that we often fail to take that into account when making predictions. In other words: too often we postulate that people will sleepwalk into a disaster. Call this sleepwalk bias.

I often use the idea of sleepwalk bias in conversations. However, what I’m pointing at here is a much more extreme example of assuming everyone is stupid about even normal everyday experiences, so I think it needs its own name. I’m calling it my "Are you presuming most people are stupid?" test.

Subscribe now

AI

There are a few claims about AI floating around that fail my test.

I was motivated to write this in response to this Time article: ChatGPT May Be Eroding Critical Thinking Skills, According to a New MIT Study. There are a few points in this article that break my rule.

Kosmyna, who has been a full-time research scientist at the MIT Media Lab since 2021, wanted to specifically explore the impacts of using AI for schoolwork, because more and more students are using AI. So she and her colleagues instructed subjects to write 20-minute essays based on SAT prompts, including about the ethics of philanthropy1 and the pitfalls of having too many choices.

The group that wrote essays using ChatGPT all delivered extremely similar essays that lacked original thought, relying on the same expressions and ideas. Two English teachers who assessed the essays called them largely “soulless.” The EEGs revealed low executive control and attentional engagement. And by their third essay, many of the writers simply gave the prompt to ChatGPT and had it do almost all of the work. “It was more like, ‘just give me the essay, refine this sentence, edit it, and I’m done,’” Kosmyna says.

What is this article actually telling us that the average person doesn’t already know? These seem to be the claims:

• If you use a talking robot to write your essay for you, you won’t learn as much about the topic compared to writing the essay yourself.

• Having a talking robot easily available to you makes you more likely to cheat on essay assignments.

Students using ChatGPT to write their essays for them aren’t stupid about what’s happening. Similar to students who just Google to find answers to homework problems, they’re aware that they’re making a trade-off between actual learning and saving time. This article is presuming that students are somehow blind to the idea that copying work from other places means they don’t actually learn. The average student isn’t like that. They make bad decisions when they cheat using talking robots, but they know what they’re doing.

Here’s another quote from the article:

The MIT Media Lab has recently devoted significant resources to studying different impacts of generative AI tools. Studies from earlier this year, for example, found that generally, the more time users spend talking to ChatGPT, the lonelier they feel.

It’s hard for me to imagine walking up to someone I don’t know and saying “Hey, spending a lot of time staring at your screen talking to a robot instead of interacting with real people can make you feel lonely. The experience itself can be somewhat alienating because the robot doesn’t feel human.” I don’t know how you could assume this is useful unless you assume the average person is really stupid. Would you feel comfortable telling a stranger this? Would you be able to say it in a way that isn’t demeaning?

To be clear, I think it’s good to conduct a lot of studies on “obvious” questions. My issue is with the reporting, not the original studies.

Another big claim that fails my test is that AI chatbots are useless. 10% of the world are now choosing to use them weekly. If they were useless, this would mean that 10% of the world is so stupid that they can’t tell that this tool they’re using every single week isn’t providing any value to them at all. There’s basically nothing else like this that people interact with regularly. You might think that social media like TikTok is bad for people, but it’s not “useless.” Users have fun or learn interesting facts or subtle social vibes from the TikTok videos they watch. You can criticize AI and think it’s net bad to use, but that’s a different claim from saying it’s useless. When I hear people say that AI chatbots are useless, it’s hard not to read it as a claim that almost everyone is incredibly stupid.

There’s too much of this way of talking in AI conversations. There are a lot of great criticisms of AI and chatbots, and real reasons to worry. I think that students cheating with ChatGPT is a gigantic crisis in education without clear solutions. But when people talk as if everyone using chatbots is incredibly stupid, and that people exposed to this technology are blind to the simple obvious trade-offs involved in specific situations, I come away with less respect for them. It seems like they underestimate the average person in a way that shows a lack of curiosity, or a tendency to steamroll other people’s experiences if they’re having a slightly different reaction to new technology.

Subscribe now

Some popular written AI x-risk arguments are way too overconfident. The authors seem to be coasting off the social status their confidence about a new tech is giving them. Writing like this gives the impression that all arguments for AI risk are based on pie in the sky wild inferences that give the speaker social sanction to override normal rules in debate, and maybe life more broadly. I think the core AI risk case is pretty simple and convincing, to the point that I don’t really understand how people can just write it off, but I also think the case is often represented badly and it’s understandable that a lot of people won’t consider it because of the sometimes goofy behavior of some of its loudest representatives. This list will be me trying to separate the goofier takes from better arguments for what I think is a real and serious risk.

I put the odds of catastrophic risk from AI in my lifetime as low, but not low enough not to worry and think about. Anything above a 1% chance of human extinction seems high enough that most people should engage with the basic arguments.

Even if you don’t buy the risk case, writing on AI risk often brings in enough interesting ideas about technology, history, nature, evolution, and the future to be worth reading.

I’m very aware this is all very speculative, which is why I’ve made this a separate post. I don’t want to imply that anything I’ve written about chatbots and the environment or how AI works should give me any credibility on risks from advanced AI. This is basically a completely separate field of inquiry that I’m truly just some guy in. My only authority here is that I think I have good taste in reading material, and I’ve been reading talking and thinking about AI risk for over a decade.

Right now this list is pretty short. I’ll be adding to it as I pull together other materials. I’m trying to be pretty picky and selective. I’ve listed the resources in the order I’d consume them.

Subscribe now

Articles

• The 80,000 Hours profiles on the big 3 AI risks (in my opinion the best places to start, period)

  • Risks from Power Seeking AI Systems

  • Gradual Disempowerment

  • Catastrophic AI Misuse

• On those undefeatable arguments for AI doom

  • Important to throw cold water on specific ways of thinking about AI catastrophe using fuzzy inferences that lead to drastic overconfidence.

• Matt Reardon’s The bone-simple case for AI x-risk

• Michael Nielson’s How to be a wise optimist about science and technology? and Notes on existential risk from artificial superintelligence

• Joe Carlsmith’s Existential risks from power-seeking AI

• Jacob Steinhardt’s series More is different for AI

• Holden Karnofsky’s Cold Takes series, which goes back and forth between takes I think are straightforward and convincing and takes I think are off the wall and goofy goober.

• Katja Grace’s Counterarguments to the Basic AI X-risk case

Classic texts

• Not AI, but related: Von Neumann’s Can we survive technology?