Discussion about this post

User's avatar
Sean Trott's avatar

This is well-argued (and a nice sequel to pt 1). Looking forward to pt 3!

A few thoughts:

1. You might’ve come across this already (apologies if you cited it and I missed), but Chen et al (2026) in TiCS has a nice summary of different positions one can take on machine understanding: https://www.cell.com/trends/cognitive-sciences/fulltext/S1364-6613(26)00077-X

Mostly helpful for organizing and taxonomizing the space of views.

2. I wrote a post about LLMs and understanding back in October 2022 (4 years ago, which is very weird), which contrasts an axiomatic rejection view and a duck test / functionalist view. More of a psycholinguistics perspective, and I’d probably write it differently now, but maybe still interesting from the standpoint of posterity (I actually think the core positions have not changed a lot): https://seantrott.substack.com/p/how-could-we-know-if-large-language

3. FWIW I think I agree that we should try to cleave off understanding from conscious experience, but I also think many people have a strong (sometimes implicit) intuition that understanding requires conscious experience. I know you argued against this in pt 1, but I suspect that’s behind a lot of resistance to claims about machine understanding. Not sure what to do about that, if anything.

4. Somewhat related to 3, I think an important dimension to this debate is why we want to have an answer to this at all. IMO the point of a psychological construct is to explain and predict behavior. So one sense in which this matters is whether viewing LLMs as thinking or understanding changes our expectations about how they’ll behave in a range of contexts in ways that matter for things like safety (I think it probably does). This is also evident in more deflationary accounts—for these to matter practically I think they have to cash out as llm behavior not generalizing to contexts where we’d think it should if they understood. I don’t think there’s really an answer here but I lean towards a coherentist account, where we ask whether the claim about understanding is justified insofar as it is consistent with a web of mutually supportive assumptions or pieces of evidence.

I appreciate all the work you’ve put into these!

XP's avatar

So much of the argument contra feels stuck in the 70s/80s: computer code as simple, legible, handmade if/then/else, so "of course" this simple deterministic branching is nothing like thought. That haunts Searle's arguments, even Penrose, and it's still implicit in so many modern critiques. Whenever someone dismisses AI as "just code" or even literally calls it "a bunch of if/then statements", this is that.

Intuitively, it's the difference between an algorithm encoding existing human intelligence as rules, and on the other hand deeper, messier computation _implemented_ through an algorithm.

That is, the intelligence of an LLM is not in its algorithmic matrix multiplication, but in its weights. Just like the intelligence of a human is not in the properties of their cells, or physics, but in their brain's neural connections.

That feels incredibly obvious to me, but the obvious rejoinder is: "Ah, so you _do_ agree it's still just code?" And it's hard to argue against that without it sounding like so many epicycles.

So I'm just going to forward those people this post instead.

13 more comments...

Ready for more?