“One of my favorite tests for this is to think of a word at least 5 letters long, and then (without looking!) try to picture where each letter is on a standard QWERTY keyboard” - I actually found this relatively easy, I just imagined typing it in my head. I’d be curious to know what the predominant experience of this mental exercise is! Twitter poll? Seems like it could be one of those “typical mind fallacies” (in either direction). Thanks as always for the clear thinking on complex topics Andy.
I think we can see from one persistent commenter here that there's a school of thought that's deeply invested in preserving the distinction between human and machine cognition for reasons that are not really about the philosophical argument developed in this first post. What that person doesn't seem to attempt is an explanation of which properties of an entirely biological structure are *necessary* for understanding. This is the crux of the matter because without that we cannot demonstrate that artificial systems *necessarily* lack them. This is a great example of how ideological conviction impedes philosophical investigation. By consistently diverting from the core question into ideological and commercial issues we have the platonic ideal of the politically motivated objection to AI. I've found watching that play out just as interesting as the original post, because anti-AI sentiment - and the myriad ways in which it manifests - is a source of fascination to me.
Really enjoyed this piece. My bachelor's philosophy thesis was on AI and theories of mind 25 years ago, and it's been fascinating watching that thinking run up against reality. This brought back an old thought on the quantum section.
I agree that if consciousness biased outcome probabilities, it would break the Born rule and allow faster-than-light signaling. But quantum mechanics has a second free input: which question gets asked, meaning what is measured and when. The formalism doesn't say what decides that. If consciousness operates there, as Henry Stapp has argued, it never touches the probabilities.
My personal working theory is that consciousness is the aperture where a many-worlds field of possibilities condenses into one experienced actuality, which then opens onto new possibilities, much like Whitehead's "concrescence." Each mind poses questions mostly about its own brain, and questions about separate systems don't interfere, so billions of minds can do this at once while sharing one world. That gives free will over our own actions (not their outcomes) without violating quantum statistics.
The cost is admitting consciousness is a genuine causal player, so physics is closed in its statistics but not in its choices. And your "interpreter" evidence is the hardest challenge for it. Still, if we take many-worlds seriously and see consciousness as how minds surf the same wave together, the universe can be deterministic as a whole while each path through it is genuinely chosen. Curious what others make of that.
What a lovely post. I'm not sure if this metaphor is still current and where it fits in your thinking (which I am still processing) but I always liked Gilbert Ryle's idea of a sports team on a field. Someone says what is baseball (I think for him it was cricket?) and you take them to a game. And you say there's the catcher and what he does is this, and there's the batter and this is what happens when you hit the batter with a pitch and so forth. And the person then asks "And what it is the position that is responsible for 'team spirit'?" And of course the idea is that if you can't point to it it must be this seperate non-physical thing that is as yet unaccounted for, but in reality this isn't a different thing it's a different lens. That's more an argument against a pure "ghost in the machine" position, but the broader idea of this thinking as a "category error" seems generally helpful.
On a more mundane note, I was doing an experiment recently where I had an LLM watch a film and describe the plot and motivations, and I thought about this critique that an LLM watching a film really couldn't understand a film because it had not experienced the emotional states in it -- it can only understand the film because it knows how other beings experience emotion. And there's something that is compelling about that. But it occurred to me that there are many neurodivergent folks who do not feel the sorts of emotions seen onscreen and do of course struggle, but ultimately if they can describe why people acted a certain way we don't say "Well, you will never understand a film." Something I'm chewing on.
Final thought. My literature professor used to tell this story about how someone had once said to him "The beauty of this story is it really captures what it is like to live in a small New England town. You can feel the author has really had this experience." He then asked, oh, you lived in a small New England town? No, the person replied. And it took the person a bit to realize that their idea of that experience was fully constructed, and this lived knowledge was primarily a coherence function on works they had previously read. Then they got really mad at him because it's very disorienting to realize that. I feel that this relates somehow.
I began reading this firmly convinced I'd disagree with the premise ("current AI models can really think and understand"), and I completed it having been pretty thoroughly convinced to agree. Kudos on the analytic approach to this question and the nicely argued thesis.
However, I think the most important question could be "Can AI models be improved to the point where they can do everything humans can do intellectually?" Can they replicate the flashes of insight and creativity that leads to new understanding of the world and new ways of dealing with it? Can they make what we experience as intuitive leaps? I don't know the answer, although I think I'd like the answer to be no, simply to keep some specialness tied to being human.
As a clinical neuropsychologist, I'd add that clinical cases offer striking evidence for many of these points. In blindsight, people with damage to the primary visual cortex can respond to visual information they're not aware of seeing. Confabulation (filling memory gaps with invented details, without any intent to deceive) and Gazzaniga's split-brain work show how readily introspection invents explanations. And in anosognosia, people can reason perfectly well while remaining unaware of their own deficits. If damaged brains can think without the Cartesian theatre, it's hard to see why that theatre should be a requirement for machines.
"How did merely physical animals at some point start accessing, affecting, and being affected by some ethereal realm or substance beyond physics? Why are they the only entities in the universe that can access this?" - There might be other entities in the universe that can access this! I feel like our knowledge is so limited that it is safer to say that we don't know
tl;dr, they tested brain activity and the activity predicted by both theories failed to happen.
It is unavoidable that these things are going to behave in ways that seem similar to how we behave. I think the biggest challenge I'd make to your overall assessment is this line:
>People will say that an AI is “just doing matrix multiplication,” as if the fact that none of those individual mathematical operations understands anything means the whole system cannot understand anything either.
The whole idea of individual parts of a system not understanding the whole system, but the system still operating, is pretty well understood and accepted. And substrate independence states that the same operations, running on a different substrate, should have the same outcome. So in theory, running the exact same operations a human mind runs, but on silicon, should have the same outcome.
However, human brains are not doing matrix multiplication. They are signalling with electrical currents and chemicals. The underlying physical activity that occurs in the human brain is totally different; we are simply simulating that with binary math. So the part that remains unclear to me is if that is far enough away from what we are doing as to make substrate independence kind of a mooth point; the substrate is not the difference.
BTW, since you've obviously been reading neuroscience, a book I enjoyed a lot I think you will enjoy that breaks down the idea of 'brain chauvinism' (that thinking/intelligence is something that happens only in brains) is Frank T Vertosick's "Mind" - https://www.goodreads.com/book/show/21941016. It demonstrates that intelligence is not confined to brains, breaks down the core pieces of what an intelligent system requires, and demonstrates other instances in nature we could argue intelligence exists (like our immune systems, or the process of evolution itself). It's a fascinating read, and would probably provide arguments that strengthen your position.
Much of the online (and offline) discourse fundamentally confuses thought, reasoning, understanding, meaning, cognition, experience, subjective experience, and self-awareness and treats them all as this big entangled thing that at once is obvious to a child ("of course it isn't really doing X, are you stupid?") and also incredibly profound, ineffable, unknowable.
But based on my remembering of sci-fi, I'd have expected the debate to be motivated by concerns about humanity, soul, conscience, what it truly means to live as a human etc. etc.
In reality, the main driver seems to be "not falling for the hype" - the deepest value of them all.
Edit; you're missing a verb: "So unless there are types of thinking that we can only as we’re having the phenomenal experience in the moment, it seems to me like the only parts of consciousness that can affect thinking are reducible to access consciousness."
Another typo, unless you're being especially salty: "Careful reasoning depends on specific complex kinds of coordination, like holding onto intermediate results, brining the relevant information together, understanding the semantic content of our words, notice conflicts, direct attention to different places, and revise mistakes."
This is a very confusing essay to me. There's a bunch of different distinctions being collapsed into a single axis in the first few paragraphs alone:
- Does cognition (gloss for "thinking" and "understanding") require phenomenal consciousness? (imo clearly not, seems like you agree)
- Is a non-phenomenological "consciousness-like" feature (e.g. access consciousness) necessary/sufficient for cognition either in humans or in general?
- Do we have a clear, functional definition of such a feature that can/does sit cleanly in an extra-subjective theory of human cognition? (imo clearly not, but seems like you disagree)
- For any such candidate definition is it plausible that current LLMs share such a feature? Is it known that they definitely do/do not share such a feature? If we don't believe such a feature is necessary/sufficient for cognition in general why does the plausibility of implementation in current LLMs have any bearing on whether we ought to describe them as have cognition?
- Does cognition require any other distinctive functional organization (e.g. memory, embodiment, recursion) either in humans or in general? If so, which are currently implemented in LLMs and which are simply computationally plausible?
- Should we expect *any* cognitive function to be computationally implausible? (imo no)
- To what extent are cognitive functions multiply realizable? To what extent are they substrate dependent? If they are substrate dependent is that for physical or non-physical reasons?
- Should we say "LLMs think" and "LLMs understand" or not? Is it just a matter of semantic preference? Is there even a reason to suppose there is a "right" answer to what we ought to say that isn't dependent on context/pragmatics? Is this a question about currently existing LLMs or about the potential for any computational system to hypothetically meet that bar?
- Are "thinking" and "understanding" specifically the sort of mental actions LLMs share with humans or is there a large class of mental analogue terms that we may or may not have to evaluate on their own and in context (e.g. 'motivation', 'love', 'regret', 'want', etc.)?
- Practically, ethically, epistemically, etc., does avoiding anthropomorphic language about LLMs cause greater errors (e.g. some avoiders seem to think LLMs can't solve novel math problems) or does endorsing anthropomorphic language case greater errors (e.g. some endorsers think it is morally required of us to engage in "preemptive" nuclear war with China)?
- Is the deciding factor "where this language implies AI has a lot of abilities it doesn’t have" or vice versa? Aren't there non-behavioral components of cognition that we might care about?
- Why should we consider "thinking" to be an anthropomorphic term? Do dogs think? Could aliens think? Does/could God think? Could the Chinese Room? Ditto "understanding".
- Is the difference between LLM cognition and your cognition of the same kind, order-of-magnitude, quality, or in other ways analogous to the difference between Einstein's cognition and your cognition?
- If "anthropomorphic language is useful because it helps us simplify, understand, and predict the behavior of our fellow incredibly complex systems we call humans", do we know if, how, and when it is similarly useful to understand LLMs? Do we know where it is particularly *not* useful for such?
There is so much fertile ground for articulating well-reasoned stances, but the only one with a proper treatment is the first question re: necessity of phenomenological consciousness.
Instead, after raising all of these questions the essay seems to suggest the crux of (most of?) them is that many people believe in a cartoonish version of Cartesian dualism(???). The intro finally ends with "a core part of how I think about the world is that most of our beliefs about our own minds and thought are useful but wildly imperfect folk theories and heuristics that get at reality about as much as saying 'the rock wants to move toward the ground.' I think it’s a gigantic mistake to assume we have total access to what’s going on in our minds, and to let folk beliefs about our minds’ specialness override any debate about them."
"We don't really understand ourselves so we might as well use anthropomorphic language to describe LLMs" cannot possibly be the main argument here. The *single biggest lesson* of psych, neuro, and the cognitive sciences is that using intuitive or convenient terminology to operationalize mental functioning absent long, careful, and often adversarial, empirical work *is always bad*. I'm sorry if that sounds "anal" of me, and I'm trying to avoid "hysterics", but claiming a scientific mantle in *favor* of recycling heuristic language across not just species but into an entirely novel type of cognitive entity is straight-up "********".
Beyond there, I feel like I must be missing a massive chunk of the essay. After the thorough debunking of the strawman folk theory, there is a totally fine, if meandering, treatment of the hard problem of phenomenal consciousness.
After that is a section that goes through a few broad theories of the functional organization of consciousness. Of the dozens of subfields of cognitive/computational psychology that lay out specific functional relations in human cognition, consciousness as an area of study has little well-established consensus theory. Why that field has any more bearing on whether we should use anthropomorphic terms about current LLMs than vision, memory, decision making, attention, emotional processing, higher-order processing, etc., is left unsaid. The only explanation for its relevance I can find is that the same four theories in the same order (+/- 1 on each end) listed as indicators for AI consciousness in a previous EA-sphere paper here (https://arxiv.org/html/2308.08708v3 Table 1). If the main argument of the essay is that consciousness has *nothing to do with AI "thinking" or "understanding"*, it's at best a pointless diversion. And yet in every single section the plausibility of machines to perform the role of consciousness is taken as evidence that should make us "more open to thinking that current AI models can think and understand." Baffling logic.
I wish I could read the essay as the other commenters seem to experience it, but I can't find it in the text.
I don’t think that’s the crux of most of them, I think the folk theory is preventing a lot of people from even considering them in the first place? I see the post as trying to give some reasons to be open to each to the point that they seem convincingly possible, and then say people sometimes shut down this conversation without thinking about it.
Ok, so this essay is 5 completely independent arguments?
1. Cognition doesn't depend on phenomenal consciousness, so lacking P-consciousness is not an individually sufficient argument against all hypothetical machine cognition (strongly agree).
> "Whether AI has phenomenal consciousness thus looks to me to be pretty unrelated to whether it can think."
2. Cognition does(?) depend on access consciousness, but that's ok because the functional features plausibly look like the kinds of things machines can do (with no argument that current LLMs do them. Disagree because there is no reason to believe cognition is dependent on access consciousness, so it is a non sequitur). You further argue several times that this is a *positive* reason to believe current(?) AI models can think and understand (strongly disagree)
> "if access consciousness is mainly about managing the flow of information (sharing it across a system, and deciding how much processing power different parts of the mind get), that sounds like something a machine can do."
> "taking predictive processing seriously as an account of perception would make me more open to thinking that current AI models can think and understand."
3. Most cognition is sub/unconscious even in humans and introspection is imperfect (strongly agree), which is important because lots of people believe they are a unified individual controlling their mind with perfect access (nakedly asserted. Disagree because human reporting of their understanding of their mind has little to do with their actual cognitive function exactly because most of it is sub/unconscious and you don't even show human reports).
4. Dropping the hypothesized "folk" theory means accepting a behavioral operationalization of understanding as flexible, coherent language use (disagree—the fact the folk theory makes inaccurate predictions of understanding requiring mental imagery(?) doesn't imply that dropping it ties one to a behavioral operationalization. It simply does not follow, so stands as assertion, which is fine for proposing a reasonable use of "understanding" but not for arguing that it is the *only* reasonable use of the term).
5. Because consciousness is not on its own a strong reason to be convinced machines cannot in theory "think" and "understand", it is appropriate to use anthropomorphic language when talking about LLM cognition (just "think/understand"? Arbitrary human mental terms? In what context? Why? I do understand that there is more of the argument to come later, but I don't see how any of this has anything to do with the targeted thesis. Even if your defense is that the argument is left for Part 2, you assert many times in this essay that it provides reasons to believe LLMs can think or understand, although you've edited out several instances of such, so the fact that it does not provide any such reasons is still outstanding).
I also acknowledge that your framing of the essay is as a "middle-ground" perspective against supposed "strict bans" against using mental language about LLMs but that is a flimsy framing. First, you do not demonstrate that anybody wants to ban mental language from being used about LLMs, and certainly not *in general* as opposed to in specific professional contexts. Second, you clearly want to make a positive case that such language is natural, reasonable, and generally appropriate for use about LLMs and you argue that explicitly multiple times throughout, so the "permissive middle-ground" is not the actual stakes. Third, your output shows positive argumentation for a stronger position before the current controversies about guidance on such language even in professional settings (e.g. this piece https://blog.andymasley.com/p/dont-treat-surface-level-metaphors predates the AP stylebook controversy by a week). If your argument is simply "there should not be an absolute ban on mental language re: LLMs" I have very good news for your side (no such bans) and very bad news about whether this essay matters at all (it doesn't in that case).
Finally, you are writing for readers of your essays, not for hypothetical opponents who shut down this conversation without thinking about it, who, by definition, are not reading it. That it continually implies such opponents:
- Have a silly "folk" theory of mind
- Could only possibly be objecting because they are physicalist
- Are censorial and irrational
...indicates that the function of this essay *in practice* (not saying it is the intent) is to convince your readers that there is not, in fact, a rich set of possible positions on this issue as I laid out in my first reply, but that they should confidently assume detractors are "non-physicalists", "folk theorists", etc. I think this is careless. To do this and *also* say "you are trying to prevent people from "shut[ting] down this conversation without thinking about it" is bordering on malpractice.
I think some of the disagreement in this area comes from the fact that the words "thinking" and "understanding" are ambiguous between inner processes and their measurable effects. In science, we're used to defining terms operationally, in terms of measurable effects; from which it follows that the inner process doesn't matter - any process that produces the right effects is okay. But I think some people have the intuition that the word meant the inner process, not the measurable effect, so they feel that some kind of sleight-of-hand has taken place. They feel that because the LLM achieves its effect through a different process (so they believe), it isn't real thinking or understanding.
For example, when you say that "understanding words is knowing what follows from using them", I interpret that operationally, but someone might say "LLMs don't *know* anything", because they regard understanding and knowing as processes, not effects.
I think the intuition you’re describing can even be fleshed out a little more - almost always, when there is a different process by which something is done, there are slightly different versions of the outward effects that manifest, particularly when you push up against strange circumstances. An electric car and a gasoline car may seem interchangeable in ordinary circumstances, but start to get different when there’s strange temperatures or lack of oxygen in the atmosphere or lack of gas stations. Sometimes the results are different enough that we want to use a different word, but sometimes we should just get over those differences.
Commenting to come back later. It looks like I mostly agree with you, but it's worth noting that the term "phenomenal consciousness" is deeply problematic. It was used inconsistently in Block's original paper, and it has been a complete conceptual mess ever since. There is no coherent or consistent distinction between "access" consciousness and "phenomenal" consciousness, and perpetuating this terminology just perpetuates the confusion. The concepts can't be clarified if the terms themselves are ambiguous.
this is an amazing read. thank you for sharing it…..thank you for writing it. thank you
“One of my favorite tests for this is to think of a word at least 5 letters long, and then (without looking!) try to picture where each letter is on a standard QWERTY keyboard” - I actually found this relatively easy, I just imagined typing it in my head. I’d be curious to know what the predominant experience of this mental exercise is! Twitter poll? Seems like it could be one of those “typical mind fallacies” (in either direction). Thanks as always for the clear thinking on complex topics Andy.
Haven't read the whole post yet, but it seems to come from a somewhat similar place as mine
https://bearlylegible.substack.com/p/the-wizard-in-the-mind
Edit: Boy it really is the platonic ideal of my post
Let's get that wizard he's no good
I think we can see from one persistent commenter here that there's a school of thought that's deeply invested in preserving the distinction between human and machine cognition for reasons that are not really about the philosophical argument developed in this first post. What that person doesn't seem to attempt is an explanation of which properties of an entirely biological structure are *necessary* for understanding. This is the crux of the matter because without that we cannot demonstrate that artificial systems *necessarily* lack them. This is a great example of how ideological conviction impedes philosophical investigation. By consistently diverting from the core question into ideological and commercial issues we have the platonic ideal of the politically motivated objection to AI. I've found watching that play out just as interesting as the original post, because anti-AI sentiment - and the myriad ways in which it manifests - is a source of fascination to me.
Really enjoyed this piece. My bachelor's philosophy thesis was on AI and theories of mind 25 years ago, and it's been fascinating watching that thinking run up against reality. This brought back an old thought on the quantum section.
I agree that if consciousness biased outcome probabilities, it would break the Born rule and allow faster-than-light signaling. But quantum mechanics has a second free input: which question gets asked, meaning what is measured and when. The formalism doesn't say what decides that. If consciousness operates there, as Henry Stapp has argued, it never touches the probabilities.
My personal working theory is that consciousness is the aperture where a many-worlds field of possibilities condenses into one experienced actuality, which then opens onto new possibilities, much like Whitehead's "concrescence." Each mind poses questions mostly about its own brain, and questions about separate systems don't interfere, so billions of minds can do this at once while sharing one world. That gives free will over our own actions (not their outcomes) without violating quantum statistics.
The cost is admitting consciousness is a genuine causal player, so physics is closed in its statistics but not in its choices. And your "interpreter" evidence is the hardest challenge for it. Still, if we take many-worlds seriously and see consciousness as how minds surf the same wave together, the universe can be deterministic as a whole while each path through it is genuinely chosen. Curious what others make of that.
What a lovely post. I'm not sure if this metaphor is still current and where it fits in your thinking (which I am still processing) but I always liked Gilbert Ryle's idea of a sports team on a field. Someone says what is baseball (I think for him it was cricket?) and you take them to a game. And you say there's the catcher and what he does is this, and there's the batter and this is what happens when you hit the batter with a pitch and so forth. And the person then asks "And what it is the position that is responsible for 'team spirit'?" And of course the idea is that if you can't point to it it must be this seperate non-physical thing that is as yet unaccounted for, but in reality this isn't a different thing it's a different lens. That's more an argument against a pure "ghost in the machine" position, but the broader idea of this thinking as a "category error" seems generally helpful.
On a more mundane note, I was doing an experiment recently where I had an LLM watch a film and describe the plot and motivations, and I thought about this critique that an LLM watching a film really couldn't understand a film because it had not experienced the emotional states in it -- it can only understand the film because it knows how other beings experience emotion. And there's something that is compelling about that. But it occurred to me that there are many neurodivergent folks who do not feel the sorts of emotions seen onscreen and do of course struggle, but ultimately if they can describe why people acted a certain way we don't say "Well, you will never understand a film." Something I'm chewing on.
Final thought. My literature professor used to tell this story about how someone had once said to him "The beauty of this story is it really captures what it is like to live in a small New England town. You can feel the author has really had this experience." He then asked, oh, you lived in a small New England town? No, the person replied. And it took the person a bit to realize that their idea of that experience was fully constructed, and this lived knowledge was primarily a coherence function on works they had previously read. Then they got really mad at him because it's very disorienting to realize that. I feel that this relates somehow.
And sorry for the long comment but it was a long post!!!!
I began reading this firmly convinced I'd disagree with the premise ("current AI models can really think and understand"), and I completed it having been pretty thoroughly convinced to agree. Kudos on the analytic approach to this question and the nicely argued thesis.
However, I think the most important question could be "Can AI models be improved to the point where they can do everything humans can do intellectually?" Can they replicate the flashes of insight and creativity that leads to new understanding of the world and new ways of dealing with it? Can they make what we experience as intuitive leaps? I don't know the answer, although I think I'd like the answer to be no, simply to keep some specialness tied to being human.
Thanks so much! High honor, and yeah will make some of this case in part 2 but I’m not positive they can rise to that level at all
As a clinical neuropsychologist, I'd add that clinical cases offer striking evidence for many of these points. In blindsight, people with damage to the primary visual cortex can respond to visual information they're not aware of seeing. Confabulation (filling memory gaps with invented details, without any intent to deceive) and Gazzaniga's split-brain work show how readily introspection invents explanations. And in anosognosia, people can reason perfectly well while remaining unaware of their own deficits. If damaged brains can think without the Cartesian theatre, it's hard to see why that theatre should be a requirement for machines.
These are interesting but the most we’re discussed in the article.
"How did merely physical animals at some point start accessing, affecting, and being affected by some ethereal realm or substance beyond physics? Why are they the only entities in the universe that can access this?" - There might be other entities in the universe that can access this! I feel like our knowledge is so limited that it is safer to say that we don't know
“brining the relevant information together” in the fourth bullet of the conclusion should be “bringing”.
This is an excellent article and I await Part 2!
Regarding IIT and GNWT, you might be interested in this article: https://www.scientificamerican.com/article/where-does-consciousness-come-from-two-neuroscience-theories-go-head-to-head/
tl;dr, they tested brain activity and the activity predicted by both theories failed to happen.
It is unavoidable that these things are going to behave in ways that seem similar to how we behave. I think the biggest challenge I'd make to your overall assessment is this line:
>People will say that an AI is “just doing matrix multiplication,” as if the fact that none of those individual mathematical operations understands anything means the whole system cannot understand anything either.
The whole idea of individual parts of a system not understanding the whole system, but the system still operating, is pretty well understood and accepted. And substrate independence states that the same operations, running on a different substrate, should have the same outcome. So in theory, running the exact same operations a human mind runs, but on silicon, should have the same outcome.
However, human brains are not doing matrix multiplication. They are signalling with electrical currents and chemicals. The underlying physical activity that occurs in the human brain is totally different; we are simply simulating that with binary math. So the part that remains unclear to me is if that is far enough away from what we are doing as to make substrate independence kind of a mooth point; the substrate is not the difference.
BTW, since you've obviously been reading neuroscience, a book I enjoyed a lot I think you will enjoy that breaks down the idea of 'brain chauvinism' (that thinking/intelligence is something that happens only in brains) is Frank T Vertosick's "Mind" - https://www.goodreads.com/book/show/21941016. It demonstrates that intelligence is not confined to brains, breaks down the core pieces of what an intelligent system requires, and demonstrates other instances in nature we could argue intelligence exists (like our immune systems, or the process of evolution itself). It's a fascinating read, and would probably provide arguments that strengthen your position.
Thank you for laying all of this out.
Much of the online (and offline) discourse fundamentally confuses thought, reasoning, understanding, meaning, cognition, experience, subjective experience, and self-awareness and treats them all as this big entangled thing that at once is obvious to a child ("of course it isn't really doing X, are you stupid?") and also incredibly profound, ineffable, unknowable.
But based on my remembering of sci-fi, I'd have expected the debate to be motivated by concerns about humanity, soul, conscience, what it truly means to live as a human etc. etc.
In reality, the main driver seems to be "not falling for the hype" - the deepest value of them all.
Edit; you're missing a verb: "So unless there are types of thinking that we can only as we’re having the phenomenal experience in the moment, it seems to me like the only parts of consciousness that can affect thinking are reducible to access consciousness."
Fixed thanks!
Another typo, unless you're being especially salty: "Careful reasoning depends on specific complex kinds of coordination, like holding onto intermediate results, brining the relevant information together, understanding the semantic content of our words, notice conflicts, direct attention to different places, and revise mistakes."
Needs an S, unle thoe are under pricey tariff: "Without a Cartesian theater guiding them from above, this look much more accessible to machines."
Should be “noticing”, “directing”, and “revising” too, no?
Fixed thanks!
This is a very confusing essay to me. There's a bunch of different distinctions being collapsed into a single axis in the first few paragraphs alone:
- Does cognition (gloss for "thinking" and "understanding") require phenomenal consciousness? (imo clearly not, seems like you agree)
- Is a non-phenomenological "consciousness-like" feature (e.g. access consciousness) necessary/sufficient for cognition either in humans or in general?
- Do we have a clear, functional definition of such a feature that can/does sit cleanly in an extra-subjective theory of human cognition? (imo clearly not, but seems like you disagree)
- For any such candidate definition is it plausible that current LLMs share such a feature? Is it known that they definitely do/do not share such a feature? If we don't believe such a feature is necessary/sufficient for cognition in general why does the plausibility of implementation in current LLMs have any bearing on whether we ought to describe them as have cognition?
- Does cognition require any other distinctive functional organization (e.g. memory, embodiment, recursion) either in humans or in general? If so, which are currently implemented in LLMs and which are simply computationally plausible?
- Should we expect *any* cognitive function to be computationally implausible? (imo no)
- To what extent are cognitive functions multiply realizable? To what extent are they substrate dependent? If they are substrate dependent is that for physical or non-physical reasons?
- Should we say "LLMs think" and "LLMs understand" or not? Is it just a matter of semantic preference? Is there even a reason to suppose there is a "right" answer to what we ought to say that isn't dependent on context/pragmatics? Is this a question about currently existing LLMs or about the potential for any computational system to hypothetically meet that bar?
- Are "thinking" and "understanding" specifically the sort of mental actions LLMs share with humans or is there a large class of mental analogue terms that we may or may not have to evaluate on their own and in context (e.g. 'motivation', 'love', 'regret', 'want', etc.)?
- Practically, ethically, epistemically, etc., does avoiding anthropomorphic language about LLMs cause greater errors (e.g. some avoiders seem to think LLMs can't solve novel math problems) or does endorsing anthropomorphic language case greater errors (e.g. some endorsers think it is morally required of us to engage in "preemptive" nuclear war with China)?
- Is the deciding factor "where this language implies AI has a lot of abilities it doesn’t have" or vice versa? Aren't there non-behavioral components of cognition that we might care about?
- Why should we consider "thinking" to be an anthropomorphic term? Do dogs think? Could aliens think? Does/could God think? Could the Chinese Room? Ditto "understanding".
- Is the difference between LLM cognition and your cognition of the same kind, order-of-magnitude, quality, or in other ways analogous to the difference between Einstein's cognition and your cognition?
- If "anthropomorphic language is useful because it helps us simplify, understand, and predict the behavior of our fellow incredibly complex systems we call humans", do we know if, how, and when it is similarly useful to understand LLMs? Do we know where it is particularly *not* useful for such?
There is so much fertile ground for articulating well-reasoned stances, but the only one with a proper treatment is the first question re: necessity of phenomenological consciousness.
Instead, after raising all of these questions the essay seems to suggest the crux of (most of?) them is that many people believe in a cartoonish version of Cartesian dualism(???). The intro finally ends with "a core part of how I think about the world is that most of our beliefs about our own minds and thought are useful but wildly imperfect folk theories and heuristics that get at reality about as much as saying 'the rock wants to move toward the ground.' I think it’s a gigantic mistake to assume we have total access to what’s going on in our minds, and to let folk beliefs about our minds’ specialness override any debate about them."
"We don't really understand ourselves so we might as well use anthropomorphic language to describe LLMs" cannot possibly be the main argument here. The *single biggest lesson* of psych, neuro, and the cognitive sciences is that using intuitive or convenient terminology to operationalize mental functioning absent long, careful, and often adversarial, empirical work *is always bad*. I'm sorry if that sounds "anal" of me, and I'm trying to avoid "hysterics", but claiming a scientific mantle in *favor* of recycling heuristic language across not just species but into an entirely novel type of cognitive entity is straight-up "********".
Beyond there, I feel like I must be missing a massive chunk of the essay. After the thorough debunking of the strawman folk theory, there is a totally fine, if meandering, treatment of the hard problem of phenomenal consciousness.
After that is a section that goes through a few broad theories of the functional organization of consciousness. Of the dozens of subfields of cognitive/computational psychology that lay out specific functional relations in human cognition, consciousness as an area of study has little well-established consensus theory. Why that field has any more bearing on whether we should use anthropomorphic terms about current LLMs than vision, memory, decision making, attention, emotional processing, higher-order processing, etc., is left unsaid. The only explanation for its relevance I can find is that the same four theories in the same order (+/- 1 on each end) listed as indicators for AI consciousness in a previous EA-sphere paper here (https://arxiv.org/html/2308.08708v3 Table 1). If the main argument of the essay is that consciousness has *nothing to do with AI "thinking" or "understanding"*, it's at best a pointless diversion. And yet in every single section the plausibility of machines to perform the role of consciousness is taken as evidence that should make us "more open to thinking that current AI models can think and understand." Baffling logic.
I wish I could read the essay as the other commenters seem to experience it, but I can't find it in the text.
I don’t think that’s the crux of most of them, I think the folk theory is preventing a lot of people from even considering them in the first place? I see the post as trying to give some reasons to be open to each to the point that they seem convincingly possible, and then say people sometimes shut down this conversation without thinking about it.
Ok, so this essay is 5 completely independent arguments?
1. Cognition doesn't depend on phenomenal consciousness, so lacking P-consciousness is not an individually sufficient argument against all hypothetical machine cognition (strongly agree).
> "Whether AI has phenomenal consciousness thus looks to me to be pretty unrelated to whether it can think."
2. Cognition does(?) depend on access consciousness, but that's ok because the functional features plausibly look like the kinds of things machines can do (with no argument that current LLMs do them. Disagree because there is no reason to believe cognition is dependent on access consciousness, so it is a non sequitur). You further argue several times that this is a *positive* reason to believe current(?) AI models can think and understand (strongly disagree)
> "if access consciousness is mainly about managing the flow of information (sharing it across a system, and deciding how much processing power different parts of the mind get), that sounds like something a machine can do."
> "taking predictive processing seriously as an account of perception would make me more open to thinking that current AI models can think and understand."
3. Most cognition is sub/unconscious even in humans and introspection is imperfect (strongly agree), which is important because lots of people believe they are a unified individual controlling their mind with perfect access (nakedly asserted. Disagree because human reporting of their understanding of their mind has little to do with their actual cognitive function exactly because most of it is sub/unconscious and you don't even show human reports).
4. Dropping the hypothesized "folk" theory means accepting a behavioral operationalization of understanding as flexible, coherent language use (disagree—the fact the folk theory makes inaccurate predictions of understanding requiring mental imagery(?) doesn't imply that dropping it ties one to a behavioral operationalization. It simply does not follow, so stands as assertion, which is fine for proposing a reasonable use of "understanding" but not for arguing that it is the *only* reasonable use of the term).
5. Because consciousness is not on its own a strong reason to be convinced machines cannot in theory "think" and "understand", it is appropriate to use anthropomorphic language when talking about LLM cognition (just "think/understand"? Arbitrary human mental terms? In what context? Why? I do understand that there is more of the argument to come later, but I don't see how any of this has anything to do with the targeted thesis. Even if your defense is that the argument is left for Part 2, you assert many times in this essay that it provides reasons to believe LLMs can think or understand, although you've edited out several instances of such, so the fact that it does not provide any such reasons is still outstanding).
I also acknowledge that your framing of the essay is as a "middle-ground" perspective against supposed "strict bans" against using mental language about LLMs but that is a flimsy framing. First, you do not demonstrate that anybody wants to ban mental language from being used about LLMs, and certainly not *in general* as opposed to in specific professional contexts. Second, you clearly want to make a positive case that such language is natural, reasonable, and generally appropriate for use about LLMs and you argue that explicitly multiple times throughout, so the "permissive middle-ground" is not the actual stakes. Third, your output shows positive argumentation for a stronger position before the current controversies about guidance on such language even in professional settings (e.g. this piece https://blog.andymasley.com/p/dont-treat-surface-level-metaphors predates the AP stylebook controversy by a week). If your argument is simply "there should not be an absolute ban on mental language re: LLMs" I have very good news for your side (no such bans) and very bad news about whether this essay matters at all (it doesn't in that case).
Finally, you are writing for readers of your essays, not for hypothetical opponents who shut down this conversation without thinking about it, who, by definition, are not reading it. That it continually implies such opponents:
- Have a silly "folk" theory of mind
- Could only possibly be objecting because they are physicalist
- Are censorial and irrational
...indicates that the function of this essay *in practice* (not saying it is the intent) is to convince your readers that there is not, in fact, a rich set of possible positions on this issue as I laid out in my first reply, but that they should confidently assume detractors are "non-physicalists", "folk theorists", etc. I think this is careless. To do this and *also* say "you are trying to prevent people from "shut[ting] down this conversation without thinking about it" is bordering on malpractice.
I think some of the disagreement in this area comes from the fact that the words "thinking" and "understanding" are ambiguous between inner processes and their measurable effects. In science, we're used to defining terms operationally, in terms of measurable effects; from which it follows that the inner process doesn't matter - any process that produces the right effects is okay. But I think some people have the intuition that the word meant the inner process, not the measurable effect, so they feel that some kind of sleight-of-hand has taken place. They feel that because the LLM achieves its effect through a different process (so they believe), it isn't real thinking or understanding.
For example, when you say that "understanding words is knowing what follows from using them", I interpret that operationally, but someone might say "LLMs don't *know* anything", because they regard understanding and knowing as processes, not effects.
I think the intuition you’re describing can even be fleshed out a little more - almost always, when there is a different process by which something is done, there are slightly different versions of the outward effects that manifest, particularly when you push up against strange circumstances. An electric car and a gasoline car may seem interchangeable in ordinary circumstances, but start to get different when there’s strange temperatures or lack of oxygen in the atmosphere or lack of gas stations. Sometimes the results are different enough that we want to use a different word, but sometimes we should just get over those differences.
Commenting to come back later. It looks like I mostly agree with you, but it's worth noting that the term "phenomenal consciousness" is deeply problematic. It was used inconsistently in Block's original paper, and it has been a complete conceptual mess ever since. There is no coherent or consistent distinction between "access" consciousness and "phenomenal" consciousness, and perpetuating this terminology just perpetuates the confusion. The concepts can't be clarified if the terms themselves are ambiguous.