The current LLMs can absolutely create both new knowledge and concepts that don't exist yet. The question is whether this knowledge and these concepts will be useful. Though humans struggle with this a lot too.
I feel like this dovetails with skepticism of AI creativity, and I'm of mixed feelings. On the one hand, certainly it seems very true that LLMs are currently less capable of some abstract mark of heightened creative drive that humans seem to often possess. On the other hand, it's not like we have a solid understanding of whether concept creation from humans actually emerges outside of our own "training data," so to speak; every time someone claims that humans can create radical new concepts ex nihilo, a proper historical analysis seems to put some doubt to the claims. Einstein had antecedents; Jackson Pollock had antecedents; even the idea of negative numbers, historically itself rather delayed in full theorizing and implementation among humans due to their conceptual difficulty, was developed in bits and pieces across time and has unclear origins prior to their first appearance in Chinese texts. If we don't have a full account of human creativity as is, how can we be certain we have a full picture of where AI creativity might go, or of how biased we are in our viewing?
If AI notices a new connection between two previously not-known-to-be-connected concepts, is that itself a new concept? I expect that many people would say no - but it'd still _useful_ for improving our conceptual understanding
I’d say most people vastly underestimate how well explored ideas and facts are. Just repacking old ideas in a new way can be seen as a massive knowledge contribution. LLMs will be excellent at this.
Concepts is an okay framing for what this isn’t, but I think concepts are a bit blurry. Insights? Intuitions? It’s hard to land.
If J. Dilla created brilliant new music by chopping, permuting, synthesizing, re-mixing samples of existing recordings -- and hint: he absolutely did -- then LLMs can create new thoughts and knowledge -- and hint: they already have.
‘Whether AI can generate new concepts’ seems like an important enough question to try an experiment to test. You used the hypothetical scenario of maybe not generating the concept of negative numbers if that never appears in its training corpus. that makes me wonder if one of these big AI labs used a frontier model to filter their full training corpus to remove all mentions, sentences, or paragraphs of the concept of negative numbers. And then they trained a new model from scratch using that modified corpus, we could then test whether it’s able to imagine what a negative number is or could be
There are two areas where I agree, and one where I’d strongly caution against.
First, the caution with Google is that I’ve found the AI Overview will “yes, and…” another AI’s hallucinations. Rather than verification, you get what I call Doppelgänger Hallucinations. https://midwestfrontier.ai/blog/doppel-ganger-hallucinations/
For the two areas where it can create new knowledge, one is “sanity checks.” There’s the Empire of AI water math you did. I ran the same numbers through K2 Think (UAE math LLM) and it figured out the liters-m3 mixup. Second, there’s creative writing. I wrote about that new paper from October about avoiding “mode collapse” https://midwestfrontier.ai/blog/better-halloween-jokes-prompts-2/
On that “yes and” to other hallucinations, I recently re-tried the test of asking ChatGPT what the seahorse emoji looks like. It didn’t go off in the spiral of attempting to use the emoji and constantly correcting itself for the maximum length of a reply, like it used to, but it still went off on a weird hallucinated set of tests where it asked me which of three images was the fake seahorse emoji (presumably implying the other two were real). The crazy thing is that when I checked back and clicked on the links where it got those images from, *all* of them were from articles saying that there is no seahorse emoji and has never been one, but that ChatGPT hallucinates that there has been!
It’s so hard to understand how something that is so good in some instances can fail so badly at even reading the titles of the articles it googles, but that’s weird alien intelligence for you!
I'm mostly agnostic as to how far the LLM paradigm alone can go, but...
- An "entirely new concept" can't really exist - you'd simply have no way of communicating it. At some level, any new concept needs to be comprised of or related to existing concepts. We do just fine as humans recombining concepts to build them into bigger and more interesting things.
- Big human breakthroughs of the imagination don't come out of nowhere. They often involve applying visual/sensory intuition in one area as a metaphor to another in some way. "Bending" spacetime, the concept of a "field" (the word refers to a 2d area), "what if chunks instead of continuous substance?" (Mendelian genetics, quanta), the very concept of putting space and time on a coordinate grid or lattice.
- Visual imagination is something LLMs really can't do right now, or at least are terrible at relative to humans. On the other hand, consider how often blind mathematicians have contributed to geometry or topology. What if sometimes visual intuition is a limitation?
The "LLMs can only remix" narrative has popped up a lot over the past few days, and it feels like doubling down on the - bad, never-really-true - stochastic parrot metaphor: "Is *too* stochastic parrot!"
I also don't share your skepticism about "new concepts". It feels like an odd, arbitrary distinction to draw that doesn't withstand scrutiny.
The concept of negative numbers did not come into the world de novo; it built on previous knowledge of arithmetic, debt, subtraction, and the notion of “nothing” or zero. If a human can "self-prompt" to develop a new concept, a frontier model can be prompted to develop it with them.
What I would concede, of course, is that LLMs don't just go about their day thinking about new concepts. The human operator selects the contextual input; depending on what that input is, new knowledge or new conceptual understandings may emerge.
So, I would offer a modified form: AI currently only develops very limited knowledge, and does very little conceptual work, without human prompting steering it toward specific outcomes. As autonomous scientific systems improve, this will change.
I mean the technical bar for understanding new concepts like this is extremely high. I certainly do not understand it. It looks like two old concepts were combined to create new integrability conditions for QFT. Does that make a new concept? Is that different from a _radically_ new concept, like a negative number, or is a negative number just combining the idea of going from "2" to "1", with "starting at 0"?
Seems philosophically fraught, and I don't know if it's a great way of getting evidence about how LLMs will be able to scale into the future, if I'm honest. I know some people are currently looking into training AIs on e.g. pre-1900s text to see if they can come up with stuff that seems clearly novel to us, retrospectively.
I don't know too much about QFT. Unfortunately, my subsubfield of STEM doesn't have a massive amount of data, afaict, or is perhaps more difficult for models to make progress on for other reasons (perhaps this is just Gell-mann amnesia though).
I don't love that article though, it seems to really belabor the point AI is going to generate purely insight and bullshit in equal measure. This is not obvious at all! Hallucination rates are going down over time. We can use proof formalizers, and are using them, to check novel insights. Remember: it is the worst it will ever be!
I don't want scientists, particularly university students deciding on what careers to pursue, and how to pursue them, to be caught blindsided by what I expect to happen. One should at least be entertaining the possibility that the crazy-train has only barely left the station.
I hadn’t realized until recently just how much OpenAI had donated to Trump, I need to do a deeper dive on the story. Right now I’m only paying for a Claude subscription from Anthropic, who haven’t materially supported the Trump campaign. Need to look into it more.
Is the ability to create truly novel concepts even necessary for human-level AI?
I hate to come across all post-modernist, but I'm pretty sure that most humans can't create truly novel concepts either. Humans _as a species_ can, sure - but the ability seems to me to mostly belong to humanity's geniuses and luminaries, whilst the rest of us seem to manage okay day-to-day without it.
I'm fairly skeptical about whether human-level AI is achievable with anything like the current paradigm, but regarding just 'novelty' specifically I'm afraid I can certainly envision a current-paradigm AI that could entirely replace myself, and most people I know*, without needing to be able to concieve of anything truly novel.
[* but not Andy Masley of course, the man's a genius.]
> AI has been trained on basically the full corpus of human text, rewarded for adhering to our conceptual universe, and punished for straying from it.
I don't think this argument really applies even to today's LLMs nevermind the whole technology.
Yes, in pretraining the model learns next token prediction and the above is more or less true. But in post-training it is rewarded for straying out and exploring, both in reasoning and in tool use. The reward is for getting to the goal, not for taking a human path to it.
The current LLMs can absolutely create both new knowledge and concepts that don't exist yet. The question is whether this knowledge and these concepts will be useful. Though humans struggle with this a lot too.
It's useful to remember that LLMs have seen and learned to predict every story and recorded instance of new concepts being developed.
I feel like this dovetails with skepticism of AI creativity, and I'm of mixed feelings. On the one hand, certainly it seems very true that LLMs are currently less capable of some abstract mark of heightened creative drive that humans seem to often possess. On the other hand, it's not like we have a solid understanding of whether concept creation from humans actually emerges outside of our own "training data," so to speak; every time someone claims that humans can create radical new concepts ex nihilo, a proper historical analysis seems to put some doubt to the claims. Einstein had antecedents; Jackson Pollock had antecedents; even the idea of negative numbers, historically itself rather delayed in full theorizing and implementation among humans due to their conceptual difficulty, was developed in bits and pieces across time and has unclear origins prior to their first appearance in Chinese texts. If we don't have a full account of human creativity as is, how can we be certain we have a full picture of where AI creativity might go, or of how biased we are in our viewing?
Another aspect here is:
If AI notices a new connection between two previously not-known-to-be-connected concepts, is that itself a new concept? I expect that many people would say no - but it'd still _useful_ for improving our conceptual understanding
I’d say most people vastly underestimate how well explored ideas and facts are. Just repacking old ideas in a new way can be seen as a massive knowledge contribution. LLMs will be excellent at this.
Concepts is an okay framing for what this isn’t, but I think concepts are a bit blurry. Insights? Intuitions? It’s hard to land.
Overestimate *
If J. Dilla created brilliant new music by chopping, permuting, synthesizing, re-mixing samples of existing recordings -- and hint: he absolutely did -- then LLMs can create new thoughts and knowledge -- and hint: they already have.
‘Whether AI can generate new concepts’ seems like an important enough question to try an experiment to test. You used the hypothetical scenario of maybe not generating the concept of negative numbers if that never appears in its training corpus. that makes me wonder if one of these big AI labs used a frontier model to filter their full training corpus to remove all mentions, sentences, or paragraphs of the concept of negative numbers. And then they trained a new model from scratch using that modified corpus, we could then test whether it’s able to imagine what a negative number is or could be
There are two areas where I agree, and one where I’d strongly caution against.
First, the caution with Google is that I’ve found the AI Overview will “yes, and…” another AI’s hallucinations. Rather than verification, you get what I call Doppelgänger Hallucinations. https://midwestfrontier.ai/blog/doppel-ganger-hallucinations/
For the two areas where it can create new knowledge, one is “sanity checks.” There’s the Empire of AI water math you did. I ran the same numbers through K2 Think (UAE math LLM) and it figured out the liters-m3 mixup. Second, there’s creative writing. I wrote about that new paper from October about avoiding “mode collapse” https://midwestfrontier.ai/blog/better-halloween-jokes-prompts-2/
On that “yes and” to other hallucinations, I recently re-tried the test of asking ChatGPT what the seahorse emoji looks like. It didn’t go off in the spiral of attempting to use the emoji and constantly correcting itself for the maximum length of a reply, like it used to, but it still went off on a weird hallucinated set of tests where it asked me which of three images was the fake seahorse emoji (presumably implying the other two were real). The crazy thing is that when I checked back and clicked on the links where it got those images from, *all* of them were from articles saying that there is no seahorse emoji and has never been one, but that ChatGPT hallucinates that there has been!
It’s so hard to understand how something that is so good in some instances can fail so badly at even reading the titles of the articles it googles, but that’s weird alien intelligence for you!
I'm mostly agnostic as to how far the LLM paradigm alone can go, but...
- An "entirely new concept" can't really exist - you'd simply have no way of communicating it. At some level, any new concept needs to be comprised of or related to existing concepts. We do just fine as humans recombining concepts to build them into bigger and more interesting things.
- Big human breakthroughs of the imagination don't come out of nowhere. They often involve applying visual/sensory intuition in one area as a metaphor to another in some way. "Bending" spacetime, the concept of a "field" (the word refers to a 2d area), "what if chunks instead of continuous substance?" (Mendelian genetics, quanta), the very concept of putting space and time on a coordinate grid or lattice.
- Visual imagination is something LLMs really can't do right now, or at least are terrible at relative to humans. On the other hand, consider how often blind mathematicians have contributed to geometry or topology. What if sometimes visual intuition is a limitation?
The "LLMs can only remix" narrative has popped up a lot over the past few days, and it feels like doubling down on the - bad, never-really-true - stochastic parrot metaphor: "Is *too* stochastic parrot!"
I also don't share your skepticism about "new concepts". It feels like an odd, arbitrary distinction to draw that doesn't withstand scrutiny.
The concept of negative numbers did not come into the world de novo; it built on previous knowledge of arithmetic, debt, subtraction, and the notion of “nothing” or zero. If a human can "self-prompt" to develop a new concept, a frontier model can be prompted to develop it with them.
What I would concede, of course, is that LLMs don't just go about their day thinking about new concepts. The human operator selects the contextual input; depending on what that input is, new knowledge or new conceptual understandings may emerge.
So, I would offer a modified form: AI currently only develops very limited knowledge, and does very little conceptual work, without human prompting steering it toward specific outcomes. As autonomous scientific systems improve, this will change.
I guess, what is a concept? Would this not count: https://x.com/hsu_steve/status/1996034524774637885
I mean the technical bar for understanding new concepts like this is extremely high. I certainly do not understand it. It looks like two old concepts were combined to create new integrability conditions for QFT. Does that make a new concept? Is that different from a _radically_ new concept, like a negative number, or is a negative number just combining the idea of going from "2" to "1", with "starting at 0"?
Seems philosophically fraught, and I don't know if it's a great way of getting evidence about how LLMs will be able to scale into the future, if I'm honest. I know some people are currently looking into training AIs on e.g. pre-1900s text to see if they can come up with stuff that seems clearly novel to us, retrospectively.
I would not hold up the Hsu paper as a good example. See this critique by Jonathan Oppenheim: https://superposer.substack.com/p/we-are-in-the-era-of-science-slop
Fair enough, does seem incorrect.
I don't know too much about QFT. Unfortunately, my subsubfield of STEM doesn't have a massive amount of data, afaict, or is perhaps more difficult for models to make progress on for other reasons (perhaps this is just Gell-mann amnesia though).
I don't love that article though, it seems to really belabor the point AI is going to generate purely insight and bullshit in equal measure. This is not obvious at all! Hallucination rates are going down over time. We can use proof formalizers, and are using them, to check novel insights. Remember: it is the worst it will ever be!
I don't want scientists, particularly university students deciding on what careers to pursue, and how to pursue them, to be caught blindsided by what I expect to happen. One should at least be entertaining the possibility that the crazy-train has only barely left the station.
Hi Andy, I've recently seen a lot of claims on my feed that ChatGPT had massively funded Trump and ICE. I'd love your input?
I hadn’t realized until recently just how much OpenAI had donated to Trump, I need to do a deeper dive on the story. Right now I’m only paying for a Claude subscription from Anthropic, who haven’t materially supported the Trump campaign. Need to look into it more.
Is the ability to create truly novel concepts even necessary for human-level AI?
I hate to come across all post-modernist, but I'm pretty sure that most humans can't create truly novel concepts either. Humans _as a species_ can, sure - but the ability seems to me to mostly belong to humanity's geniuses and luminaries, whilst the rest of us seem to manage okay day-to-day without it.
I'm fairly skeptical about whether human-level AI is achievable with anything like the current paradigm, but regarding just 'novelty' specifically I'm afraid I can certainly envision a current-paradigm AI that could entirely replace myself, and most people I know*, without needing to be able to concieve of anything truly novel.
[* but not Andy Masley of course, the man's a genius.]
Pretty much agree!
I like this framework of knowledge vs. concepts; it's new to me but very functional
> AI has been trained on basically the full corpus of human text, rewarded for adhering to our conceptual universe, and punished for straying from it.
I don't think this argument really applies even to today's LLMs nevermind the whole technology.
Yes, in pretraining the model learns next token prediction and the above is more or less true. But in post-training it is rewarded for straying out and exploring, both in reasoning and in tool use. The reward is for getting to the goal, not for taking a human path to it.
Yes, LLMs generate new knowledge constantly.
But conceptual innovation is a coordination problem, not a compute problem.
It requires shifting not just beliefs, but the shared structures that make beliefs legible.
That’s why models feel powerful yet constrained.