This post won’t have arguments for why we should legalize Waymo. I have a separate post on that, though I’d like to write an update to it since I think there are much stronger cases I can make.

You may have noticed that Waymos are already driving around DC (with humans at the wheel). They’ve been mapping the city and say they’re nearly ready to move forward, but the city currently has no legal path to commercial service. Two decisions are in the way: a driverless testing pathway and a commercial operating framework. Right now both are held up by a missing District Department of Transportation (DDOT) report that was supposed to arrive in October 2022, and by DDOT’s failure to open a permit pathway separate from the report that already exists in statute.

Contents

• What’s blocking Waymo

  • The 2 things Waymo needs

  • The DDOT report and the separate testing permit framework

  • The council and mayor debate once the report is published

• The mayoral race

  • Kenyan McDuffie

  • Janeese Lewis George

  • The mayoral race makes the DDOT timeline much more important, and vice versa

• The federal government

• How to legalize Waymo in DC

  • The most promising levers to pull

  • Lessons from other cities

What’s blocking Waymo

The 2 things Waymo needs

Waymo needs two things from DC’s city government that it doesn’t have.

• Driverless testing permits. Right now, every Waymo in DC is required to have a human safety driver. But Waymos need to be tested without a driver. This has been a standard step in every other city. DC law already authorizes DDOT to run an AV testing permit program that can include remote-operator (driverless) testing, and 2025 amendments specify that driverless testing requires that permit. But DDOT has never made the application for driverless permits available. Until DDOT opens the permit pathway and finishes the associated rules, testing can only continue with a human physically present, and the necessary driverless tests can’t happen.

• A commercial operating framework. Even if Waymo got driverless testing permits tomorrow, current DC law has no structure for a commercial autonomous ride-hailing service. Things like insurance requirements, licensing structures, and rules for how a driverless vehicle interacts with police don’t exist. Autonomous ride-hailing is effectively illegal in DC.

For Waymo to operate normally, they need to get driverless testing permits, then begin commercial operations under a legal framework the council has passed. Neither step is available today.

The DDOT report and the separate testing permit framework

The Autonomous Vehicles Testing Program Amendment Act of 2020 gave DDOT authority over AVs and required the department to produce a report within one year of the law's applicability date of October 1, 2021, meaning the deadline was October 1, 2022. The report was to provide “recommendations to safely accommodate the deployment of autonomous vehicles on public roadways for commercial, personal, and any other use.” The report could optionally include draft legislation or regulations.

This report is more than three and a half years late, and is now the main political obstacle blocking Waymo.

The 2020 law tasked DDOT with setting up a formal AV testing program as a distinct obligation from the deployment report. DDOT hasn't indicated they're working on permits separately, so we may not get clarity on testing permits until the report comes out, if it addresses them at all.

It blocks legislation because Charles Allen, the DC Council’s transportation committee chair, has made the finished report a precondition for holding hearings on any AV bill. Former councilmember Kenyan McDuffie (one of the two leading candidates for mayor) introduced a comprehensive AV legalization bill (the Autonomous Vehicles Amendment Act, B26-0323) in July 2025. It’s been sitting in Allen’s committee ever since without a hearing, because Allen won’t schedule one until the report is finished. Nothing in the law says the council can’t legislate on AVs without this report. Allen’s position is a political choice.

The history of the report

• December 2020: Autonomous Vehicles Testing Program Amendment Act of 2020, giving DDOT authority over AVs and required the initial report. The entire mandate says:

  • “Within one year after the applicability date...the Department shall transmit to the Council a report that provides recommendations to safely accommodate the deployment of autonomous vehicles on public roadways for commercial, personal, and any other use the Department determines. The report may include draft legislation or regulations.”

• January 2022: DDOT publishes a notice of proposed rulemaking in the DC Register to create an AV testing program. This never gets finalized. This is a separate obligation from the report, more on this below.

• Late 2022: The report’s deadline of October 1, 2022 passes with no public explanation. I can find no reporting on what, if anything, DDOT was doing on the report during this period.

• 2023: No report, no public explanation for the delay. There’s no reporting anywhere I can find on the reason it was delayed.

• July 2024: A Wayback Machine snapshot of the mayor’s AV Working Group’s webpage (which has since been removed from the DMOI website) reveals that DDOT was still in the procurement phase for hiring a consultant to help define a regulatory framework. More than two years after the report was due, the work of writing it had not yet been contracted out.

• March 2025: Congress freezes nearly $1 billion in DC’s local funds. Allen later attributes part of the report’s delay to this, but the report was already almost three years late at this point.

• Spring 2025: Some expected the report to be finished around this time. It did not materialize.

• Fall 2025: DDOT told Allen the report would be ready around this time. In September, DDOT backtracked, telling Allen it would be out “in the new year, timeline unknown.” Allen said he had been meeting with trial lawyers, safety advocates, and Waymo in preparation.

• Late 2025: DDOT said the report was one of the programs affected by budget cuts. But this doesn’t explain why it wasn’t finished in 2023 or 2024, before the cuts happened. DC has since entered a new fiscal year and the funds have been restored.

• February 2026: A DDOT spokesperson told Axios the agency expects to propose recommendations “later this year.” Waymo told Axios that DDOT informed the company the report “is being actively worked on, and could be finished by this summer.”

Will the report be finished by summer? I’m skeptical. But there are three reasons to think this time could be different. Political pressure is higher than it’s ever been. The Washington Post editorial board has now published two pieces on this (one literally accusing DDOT of “sabotage”), Waymo is spending a lot more on lobbying, and the company got 1,500 residents to contact city hall. Also, DDOT appears to have finally contracted outside consultants to help write it. And the fact that DDOT gave a specific timeframe to both Axios and Waymo, rather than the usual vague non-answers, suggests at least something is happening.

What happened with that AV testing program?

DDOT published proposed rules in January 2022 to create the AV testing program. This is separate from the report. The 2020 law created two distinct obligations: a testing permit program and a deployment report. The proposed rules would have established how companies apply for testing permits and what the testing framework looks like. If DDOT had finalized them on any reasonable timeline, Waymo could have applied for driverless testing permits years ago.

DDOT never finalized them. The mayor’s now apparently dormant AV Working Group’s own page, as late as mid-2024, said the rulemaking was “still in the works” and they looked “forward to keeping the public informed when there is more to share.”

This is in my opinion more ridiculous than the delayed report. DDOT got far enough to publish draft rules for driverless permits and then just stopped. They should really be held accountable here. Waymo could begin driverless testing with permits while they wait for legislation to be fully legalized if DDOT could just get its act together on this separate thing, but instead they’ve gone completely silent on it.

This created an obvious problem when Waymo and other companies started showing up to test in early 2024. There was no permit system, so there were no formal rules. The council responded to this problem with the Autonomous Vehicle Testing Permit Requirement Amendment Act of 2024 (B25-0710), which Allen’s committee held a hearing on in June 2024. The law was simple: until DDOT’s permit program actually exists, driverless testing is prohibited. Any company testing AVs must have a human in the vehicle and give DDOT advance notice.

Because temporary legislation in DC expires and must be renewed, this had to be passed over and over, as an emergency measure in March 2024, a temporary law in June 2024, another temporary law in March 2025, and a congressional review emergency act in March, before finally being made permanent through D.C. Law 25-312 in the same month. DDOT’s inaction forced the council into a patchwork of reactive legislation instead of operating under the coherent framework that was supposed to exist by 2022.

Even if the deployment report appeared tomorrow, there would still be no finalized testing permit framework from DDOT. The report and the testing rules are separate obligations, and both are unfinished. Anyone who tells you the report is the only thing blocking Waymo is missing half the story. This is something I think a lot of AV advocates are missing in conversations about Waymo in DC.

It’s worth noting that the report could end up addressing testing permits. The mandate says the report “may include draft legislation or regulations,” so DDOT could bundle everything together. But there are reasons to think it won’t:

• The 2020 law created them as separate obligations. The testing permit program was established under a different section of the law than the deployment report. DDOT published proposed rules for the testing program in January 2022, a completely separate regulatory track, and then abandoned them. The fact that DDOT initiated a standalone rulemaking for testing permits confirms that the agency itself originally treated these as distinct workstreams.

• DDOT has never said the report will include testing permits. Despite nearly four years of delay, DDOT hasn’t publicly indicated that the report will address the testing permit framework. If they were planning to combine the two, you’d expect them to say so, especially given how much scrutiny this issue is getting.

• The report and the permits are controlled by different bottlenecks. Allen’s precondition is about legislation: he won’t hold hearings on AV bills until the report is done. But testing permits are an executive function. DDOT could issue them under existing statutory authority without any new legislation passing. These are different problems held up by different branches of government, which makes it less likely that a single document resolves both.

What the report might say

The DDOT report’s content matters a lot. It’s possible that the report recommends heavy restrictions, and that would obviously be terrible for the prospects for AVs in the city anytime soon. There are maybe three possibilities:

• A permissive framework would recommend clear pathways for testing and commercial AVs with standard requirements. This is basically what McDuffie’s bill already proposes. It would give Allen what he says he needs, and things could move pretty quickly.

• A cautious, phased framework might recommend a limited testing period, geographic restrictions, fleet size caps, mandatory human oversight periods, and extensive reporting requirements before any expansion. This would still move things forward but could add a year or more.

• A restrictive or inconclusive framework is the worst case. If the report emphasizes unresolved safety questions, calls for further study on labor impacts, or recommends conditions Waymo can’t practically meet (like requiring a human in every vehicle), it gives official cover to delay indefinitely.

A couple things would really help AV advocates: if the report addresses testing permits alongside deployment (rather than leaving those stuck in a separate bureaucratic track), and if it includes draft legislation (as the mandate allows), which could speed up the council’s work significantly.

The Three Gatekeepers

Three parties share responsibility for this block: DDOT, Charles Allen, and Mayor Bowser.

DDOT

DDOT is more than three and a half years late on the report. The agency has a lot on its plate, and when it received this assignment AVs were much more speculative.

Some people have speculated about internal opposition to AVs at DDOT, but I don’t see strong evidence for this. DDOT Director Sharon Kershbaum hasn’t publicly commented on AVs. Some have inferred anti-AV sentiment from her other unrelated positions that they read as anti progress, but I think that’s a stretch. Transit decisions are complicated and don’t map neatly onto a pro- or anti-tech progress axis. I’d identify as pretty pro-progress, but I share Kershbaum’s lack of enthusiasm for dedicated bike lanes (sorry friends), and I don’t think that means I’m anti-progress on other transit issues.

Stephanie Dock is the manager of DDOT’s Autonomous Vehicles Program and leads the team responsible for overseeing autonomous vehicle testing. She was on a panel on AVs presented by MIT in 2025 and had some useful quotes on the report:

We want to be adaptable, we want to be responsive to what’s going, and the industry is changing a lot faster than our rulemaking structure is allowing us to keep pace.

In order for us to do a permitting program, we have to have done a rulemaking.

Trying to figure it out before it comes. Anybody else who’s struggling with that prospect, you are not alone. We are also trying to figure that out.

These are notable because they’re the closest thing to a public explanation for why DDOT has been slow. She’s essentially saying the bureaucratic process itself (rulemaking requirements) is a structural bottleneck, not just a prioritization failure.

On the deployment report:

A deployment report with recommendations for legislation and regulation... that has to be gone through a lot of signatures, very similar regulations, so I have no promise on when the full report will be released, but I... fingers crossed, we’ll have made it through the procurement process so that... such that we can release the research aspects, kind of the background work.

This confirms the report was still going through procurement/signatures as of this talk in 2025, and that she herself couldn’t promise a timeline.

On industry vs. DDOT’s timeline:

Industry has slightly different plans. I am still looking at recommendations for deployment. Industry is certainly saying we would like to deploy. So they said they’re coming in 2026. I am eagerly awaiting how they’re going to figure out to do that.

But they also have political levers that they can pull that I, as a lowly staffer, do not.

So here it seems like she’s openly acknowledging that Waymo’s 2026 deployment plan doesn’t align with where DDOT actually is, and that the company has political tools (lobbying, public pressure) that she doesn’t.

On DC’s attractiveness and risk for AV companies:

I think everyone knows that the risks of coming into our market are relatively high, right? You’ve got to do it right, because if you want to convince Congress that you need less oversight of the AV industry, you better not be screwing up their travel on a daily basis.

On the depot/workforce concern:

Worried that if we are unable to bring in some of the maintenance and charging aspects of these fleets, that any of the workforce benefits that might accrue associated with this are actually going to go to us, because they’re going to go to Maryland and Virginia, where there is more land.”

“It also has a risk of increasing the zombie miles of empty VMT driving back as they deadhead to get to those facilities.

This shows she’s thinking about empty miles and workforce displacement to the suburbs. She’s not anti-AV here, but she’s clearly concerned about the impacts.

On where they actually are (no deployment framework):

You’ll note that my slides here with the timeline of where we are in legislation and regulation makes absolutely no mention of deployment. We do not have legislation for that at this time.

On the regulatory vs. investment mindset:

I spend a lot of time thinking about what we regulate. I get to spend some time thinking about what we incentivize, and I spend very little time thinking about what we could invest in. I’d like to change that eventually, but first we have to have these base frameworks in place.

She also brings up DC’s objectively weird transit situation:

Lots of places have motorcades. I would argue that we probably have more than most, and they look in all different sizes... it could be everything from the president or the vice president moving on street... but also, we run a lot of unannounced motorcades that are just a couple vehicles moving through, as perhaps a senator is coming up to Congress. There’ll only be two or three black cars running all sorts of red lights.

We are the central city in a tri-state region... on the one hand, we are a state, but at the same time, we’re also at the very middle of a region, and our region involves Maryland and Virginia, and depending on the commute shed that you’re looking at goes as far as West Virginia as well.

DC does not... is not an industrial city at all, right? That’s not why we exist. So we have a real shortage of land that can be used for anything like a depot. We don’t have a lot of big warehouses.

DDOT seems to have expanded the scope well beyond what the law required. The July 2024 Wayback Machine snapshot of the AV Working Group page says DDOT was trying to “procure consultant support to help the agency define a potential regulatory and legislative framework for the deployment of AVs.” That’s already broader than “recommendations” the original law called for. And Allen has added on additional expectations. In his public statements he’s talked about wanting the report to address job displacement, worker transition, and what he calls getting it “right” in a way that clearly extends beyond the safety focus of the original mandate. His December 2025 newsletter explicitly raised “the impact on jobs” as something the report needs to address. The law never called for that.

It’s crazy that after almost four years, we don’t know for sure if DDOT is planning to actually provide a permitting system along with the report. It would be nice if they would just make the general goals for the report clear, because even there we have very little to go on besides the original very short mandate.

Charles Allen

Allen is the single most important person on the legislative side. He’s the chair of the Committee on Transportation and the Environment, so he controls whether any AV bill gets considered. His making the DDOT report a precondition has given the department a total veto over the legislative timeline.

Allen has been careful to say he’s not opposed to AVs. He’s said he doesn’t “think we should be afraid of this technology or throw up obstacles” and that autonomous vehicles will “likely operate in the District” in the near future. But the practical effect of his position is indefinite delay.

His stated reasons are about process and safety. From his December 2025 newsletter:

The idea that we should authorize fully driverless vehicles without DDOT’s analysis in hand is not safe, nor is it even guaranteed to happen if the Mayor and DDOT don’t want to issue permits.

He’s also raised job impacts:

The questions also need a stronger conversation about the impact on jobs — DC residents working as taxi and rideshare drivers today would be replaced, and the money you pay for a ride would no longer go back into the community but rather be sent out of state to corporations largely located in California.

That one sounds especially bad, because there’s no way that Waymo (or buses, or bikes) could be legalized without reducing the number of rideshare jobs at least a bit.

Some other key quotes from Allen on how he’s thinking about this:

Autonomous vehicles hold promise, but DC residents should not be treated like test subjects while companies work through unresolved issues on busy city streets. Given recent nationwide examples of AVs making dangerous mistakes, a careful, data-driven approach is warranted. The questions also need a stronger conversation about the impact on jobs – DC residents working as taxi and rideshare drivers today would be replaced, and the money you pay for a ride would no longer go back into the community but rather be sent out of state to corporations largely located in California.

We’re committed to getting this right – advancing innovation without compromising safety – and to making sure autonomous vehicles are integrated thoughtfully into a transportation system that works for everyone.

- December 2025 newsletter

I don’t think we should be afraid of this technology or throw up obstacles. But my job isn’t to say, ‘How fast can I roll this out?’, but rather, ‘How can we get this right?’ I’d rather get it done right than get it done fast.

- Interview with The 51st

(in response to a claim that AVs are involved in more crashes than human driven cars) This to me is a great example that I don’t necessarily want to be the first into this marketplace. I don’t mind having other cities go first but let’s learn what works and what doesn’t and then figure out what’s best for D.C.

- Interview with FOX 5 DC

Could he change his mind on making the report a condition? Nothing legally prevents him from holding hearings without the report. McDuffie’s bill exists and could be brought to hearing. But Allen has made this a matter of principle, and I’m skeptical that the relatively small number of AV advocates in DC can change his position on it. I also personally just have no way to judge how reasonable his decision is. There’s a legitimate case that the council should have expert analysis before writing complex regulatory legislation. What I can say is that DDOT’s three year delay has turned what might be reasonable caution into an effective block. I do worry a lot about a lot of the quotes he’s shared here on safety and jobs. Both go way beyond a reasonable or fair assessment of Waymo compared to any other vehicles.

Allen has no control over DDOT, but someone else does…

Mayor Bowser

Mayor Bowser also bears responsibility here. DDOT is an executive agency that reports to the mayor. The report is almost four years overdue on her watch. Bowser could direct DDOT to prioritize it. She could also direct DDOT to separately begin designing and issuing driverless testing permits.

Allen has publicly blamed Mayor Bowser for not issuing driverless testing permits. In December 2025, he told Axios “I don’t know what the holdup is.” Bowser’s office responded to a media request on this anonymously. An official who wasn’t authorized to speak to the press told Axios “It’s not us” and characterized DDOT as “being super diligent with their research.” I’m not sure what to infer from this.

The fact that DDOT has been allowed to sit on this for nearly four years without consequence suggests either that Bowser doesn’t consider AVs a priority, or that there’s an active decision to keep this in limbo.

Bowser could unblock the situation from the executive side without needing the council to do anything. Getting DDOT to produce the report would remove the justification Allen is using to hold up legislation.

Bowser has less than a year left of being mayor after 3 terms. A new mayor will take office in January of next year, and the primary that will determine who will be mayor is happening this June. Bowser hasn’t given any sign that she’ll give any priority to AVs in her last year as mayor, so while I wish she’d use her executive authority to push DDOT to prioritize the report more and to separately issue driverless testing permits in the meantime, I don’t expect this to happen. She doesn’t have much incentive to push something new and controversial like this.

Takeaways

• Waymo needs two things from DC’s government to operate: testing permits for Waymos to drive without humans inside, and legalization of commercial AVs.

• The report is the main political bottleneck for both. It blocks legislation because Charles Allen, the transportation committee chair, won’t hold hearings without it. It’s also part of the justification DDOT gives for not moving forward on testing permits, even though the 2020 law established testing permits and the deployment report as two separate obligations, and it’s not even clear the report will address testing permits at all.

• But the report isn’t the only bottleneck. DDOT’s unfinished permit and rulemaking process (a separate track from the report) is the practical bottleneck for driverless testing. The driverless testing permits could also be produced if Mayor Bowser decided to direct DDOT to open the permit pathway that already exists in statute, but it looks like she won’t, and DDOT hasn’t given any indication that they’re separately working on driverless testing permits.

• There are recent signs of progress. DDOT told both Axios and Waymo the report could be done by summer 2026, and they appear to have contracted outside consultants. But DDOT has missed deadlines before and is more than three and a half years late.

The council and mayor debate once the report is published

Once the report is published, this will only clear the way for the DC council to actually make policy on AVs, and we know almost nothing about what each DC councilmember believes. Many other cities in the Northeast have delayed or rejected self-driving cars.

Of the 13 councilmembers, the only one besides Allen who has been publicly vocal about Waymo is Janeese Lewis George, and she’s been skeptical of AVs (more on her below, note that she’s the other main candidate for the next DC mayor). Everyone else seems to not have commented at all. I cannot find any hints of what any of the councilmembers think about Waymo.

One piece of info we do have is who voted for the 2020 AV bill that created the original AV framework and gave DDOT authority over AVs. The bill seems pretty pro AV, it was passed unanimously, and 7 of the 13 councilmembers who passed it are still in office, though some are not running for re-election and will be out of office in January 2027:

Current councilmembers who voted for the 2020 bill:

• Phil Mendelson

• Anita Bonds (not running for re-election)

• Robert White (running for Congress)

• Brianne Nadeau (not running for re-election)

• Brooke Pinto (running for Congress)

• Charles Allen

• Trayon White (currently awaiting a federal bribery trial scheduled for September, creating an unprecedented dynamic where he was unanimously expelled from the DC Council in February 2025, only to win his Ward 8 seat back in a July 2025 special election)

The six councilmembers who weren’t in office in 2020 are:

• Janeese Lewis George

• Christina Henderson

• Doni Crawford

• Matt Frumin

• Zachary Parker

• Wendell Felder

If a councilmember hasn’t thought about AVs, this might be a great opportunity for DC citizens and groups to influence their opinion early before the issue becomes polarized, or before the anti-AV interest groups reach them first. This actually seems really urgent in a way that hasn’t been covered as much, because even when the report is passed, the decision to legalize Waymo will be in the hands of these 13 councilmembers, and the only one who’s spoken about Waymo specifically has been negative. I think we should actually be somewhat pessimistic about how the council will vote. As noted in Understanding AI, Waymo has had a lot of success in Republican-dominated states where the regulatory environment is more favorable, and has hit barriers in more Democratic states. As a baseline, because DC skews very left, and this pattern of delay in more left of center places is becoming more consistent, we should expect a higher chance of bumping into more political opposition here.

One other note is that the two council votes running for Congress (Pinto and White) will be heavily influenced by how organized labor (a vital constituency in a citywide/district-wide congressional primary) views the AV issue.

The mayoral race

I really don’t think people should vote for DC mayor based on how they think about AVs, but it’s also important to understand that the two main candidates for DC mayor seem to be on opposite sides of the AV debate and it’s likely that this election will be extremely consequential for the future of self driving cars in the city.

The mayoral race in DC is basically decided by the Democratic primary on June 16 2026. Because DC is overwhelmingly Democratic, whoever wins that primary will almost certainly be the next mayor, taking office in January.

The two leading candidates in the primary are Janeese Lewis George and Kenyan McDuffie. McDuffie was the most pro-AV member of the DC council, and Lewis George is the most anti-AV.

Here’s a prediction market for who will win:

Kenyan McDuffie

McDuffie is the former councilmember who introduced the Autonomous Vehicles Amendment Act of 2025 (B26-0323), the bill that would create DC’s complete AV operating framework. The bill was introduced on July 11, 2025 and referred to the Committee on Transportation and the Environment on July 14, 2025. It would allow fully autonomous vehicles to operate without a human driver; establish insurance and financial responsibility requirements; require companies to submit law enforcement interaction plans; create an on-demand AV network under existing for-hire transportation law, with human-driver-only provisions carved out; and designate DDOT as the sole agency implementing the act.

McDuffie was regularly described as one of the most “pro-business” councilmembers. If elected mayor, he would likely direct DDOT to finish the report on a hard deadline, possibly issue driverless testing permits independently, and use his relationship with Allen and other councilmembers to push for hearings.

Surprisingly, McDuffie hasn’t done or said anything else related to AVs that I can find.

Janeese Lewis George

I want to flag that I’m going to be pretty critical of Lewis George’s coalition’s attitude toward AVs here, but I’m otherwise unsure of who I’m going to vote for and friends I trust have wildly different takes on which candidate is preferable in general, though basically everyone I know agrees that we’d like Lewis George to change her mind on AVs. Among other things Lewis George got the endorsements of prominent housing groups in DC who I tend to agree with.

Lewis George is the councilmember who has been most publicly skeptical of AVs. She said in a podcast interview in February “I don’t think our city is ready for Waymo at this moment.” She’s framed her concern as primarily about safety, but she also spoke about worker displacement, saying “we’re gonna create a balanced approach” because “I want to make sure we are not displacing workers.”

Lewis George describes herself as a democratic socialist and has built her career around labor advocacy. In December she got early endorsements from five unions with significant local organizing presence: SEIU 32BJ (which represents property service workers including janitors, security officers, and airport workers), ATU Local 689 (transit workers), UFCW Local 400 (grocery, retail, and healthcare workers), and Unite Here Locals 23 and 25 (hospitality and food service workers). At that endorsement rally, she pledged that as mayor she would “take on companies” engaged in union-busting and protect workers from displacement fueled by artificial intelligence. The Working Families Party and Washington Teachers’ Union have also endorsed her.

Axios reported that “a prominent local labor union” endorsed Lewis George in part because of concerns about job losses from autonomous vehicles specifically. The Washington Informer identified this union as SEIU 32BJ (property service workers), whose executive vice president Jaime Contreras said they endorsed Lewis George because "She's anti-Waymo, which is something that we oppose coming to the city because it's going to displace working-class people who do Uber or Lyft or DoorDash…for extra income.” This mirrors the pattern in other cities where organized labor has emerged as the primary opposition force to Waymo’s expansion.

Lewis George hasn’t called for banning Waymo or said autonomous vehicles should never operate in DC. Her stated position is that the city isn’t ready “at this moment” and that she wants to ensure workers aren’t displaced. This sounds reasonable, but is actually quite extreme as a reaction to new vehicles. Any rule that says we can’t introduce a new transit technology if it threatens to cause rideshare drivers to lose their jobs is an abysmal way to think about transit policy. Among other things, if buses didn’t exist, this same rule would ban buses. This same rule would have also banned rideshare apps in the first place, because they mostly destroyed the (better regulated, better paid, easier to organize) taxi industry in DC1. I have more on that argument here.

Lewis George’s position on Waymo closely matches the positions of leaders in cities that have stalled or effectively banned it:

• In Boston, the city council’s proposed ordinance uses identical framing: study the impact on jobs first, require a human safety operator in every vehicle, and don’t allow commercial operations until an advisory board gives the green light. That advisory board would be required to include representatives from the App Drivers Union (a rare example of unionized rideshare drivers), the Greater Boston Labor Council, the Teamsters, and the UFCW. Eight city councilors collaborated on the ordinance. The Teamsters organized a coalition called “Labor United Against Waymo” which was the first mass-mobilization effort outside California to push back against AV expansion. Waymo’s spokesperson called the proposal the “first major city in the world to ban fully autonomous vehicles based entirely on vibes.”

• In New York, Governor Hochul withdrew her robotaxi legalization proposal in February after failing to secure legislative support. Her spokesperson said “Based on conversations with stakeholders, including in the legislature, it was clear that the support was not there to advance this proposal” which in practice meant that organized labor made the political cost too high in an election year. The proposal would have allowed limited robotaxi deployment upstate while leaving New York City decisions to the mayor and city council. It died anyway. Waymo has testing permits for a handful of vehicles in Manhattan, but can’t charge passengers or operate commercially without new legislation that now doesn’t exist.

• In Minnesota, SEIU chapters started organizing against AV legislation before Waymo even announced plans to expand to the Twin Cities. By the time Waymo began testing, there was already organized opposition pushing for legislation that would require human drivers in every vehicle. Multiple Minneapolis city councilmembers have signaled interest in passing a citywide ordinance with the same requirement.

• In Washington state, law allows testing but requires a human safety operator in all vehicles and doesn't offer a clear path to commercialization. Seattle has its own additional testing permit program on top of the state requirements.

The same pattern shows up again and again where politicians express openness to AVs in principle, insist on studying job impacts first, and structure the study process so that it can never actually conclude that it’s okay to proceed. The “study jobs first” framing often functions as a ban.

Lewis George hasn’t gone as far as Boston’s councilors, who proposed requiring a human in every vehicle. But her framing seems to track the same playbook, her coalition includes the same types of unions driving opposition in those other cities, and the political incentives point in the same direction. The Teamsters’ national position is a permanent human-operator requirement for all autonomous vehicles. ATU Local 689, which endorsed Lewis George, has lobbied against AVs alongside its national chapter.

That said, Lewis George does have room to eventually support AV deployment. She could condition it on workforce transition programs, labor protections, or union consultation requirements. There’s a version of AV legalization that includes strong worker provisions and that she could plausibly support without reversing her position. The question is whether she’d actually pursue that as mayor, or whether the political incentives from her labor coalition would push her toward indefinite delay.

This doesn’t mean AVs would never happen under Lewis George. She hasn’t called for a permanent ban, and external pressure would keep building: Waymo would be operating in 15+ other cities, Baltimore might have the service 40 miles away, federal legislation could change the landscape. But nothing in her current positioning suggests she’d prioritize this, and the most likely outcome is that legalization gets pushed back by multiple years.

The mayoral race makes the DDOT timeline much more important, and vice versa

The mayoral race matters for AVs, but how much it matters depends almost entirely on whether the DDOT report comes out before the new mayor takes office in January 2027.

If the report comes out before January 2027, the mayoral race becomes potentially less decisive. Allen’s stated precondition would be removed. The McDuffie bill already exists. Allen could hold hearings in the fall. The council could potentially vote before the end of the year. If legislation passes before January, the new mayor inherits a legal framework, and unwinding legislation the council has already passed is much harder and more politically costly than just preventing it from ever being introduced.

Bowser could also use the report as cover to issue driverless testing permits in her final months. If DDOT hands her a finished product, the political calculus changes. She could claim credit for moving the city forward on her way out.

If the report still isn’t finished by January 2027, the mayoral election becomes the single most important variable in whether Waymo is legalized in DC anytime soon. The next mayor controls DDOT, and the report’s absence is what keeps everything frozen. Under Lewis George, the stall could extend for years, especially if she adds requirements to look into effects on jobs and halts legalization until there are guarantees that rideshare drivers won’t lose work.

Under McDuffie, things would likely move faster, though he’d need to navigate the council’s labor concerns.

This is why pushing for the DDOT report to come out before the election is the single most important thing AV advocates can do. Every month of delay past this summer makes the mayoral outcome matter more.

The federal government

Congress has direct authority over DC. While the local political dynamics I’ve described above are the most important near-term variables, federal action could eventually change the landscape, though right now there are no especially promising bills. Two bills that would in part pre-empt DC’s decision making on AVs are the SELF DRIVE Act of 2026 and the Autonomous Vehicle Acceleration Act of 2025 (S.1798), but both look very unlikely to become law. The SELF DRIVE Act is being talked about as maybe possibly being folded into the Surface Transportation Reauthorization, which would make it more likely to pass, but this is all pretty speculative. Technically the federal government could eventually pre-empt DC law if they passively disallow or actively ban Waymos, though I wouldn’t count on it anytime soon.

How to legalize Waymo in DC

The most promising levers to pull

I don’t know a lot about the specifics of influencing city government, so take these with a grain of salt. I’ll update them as I get feedback. The three main things to do right now seem to be:

• Apply public pressure to DDOT to finish the report on time, and then to create the separate necessary testing permit framework. This could be through public writing or writing to representatives.

• Make a plan for how to sway the councilmembers who haven’t expressed any opinions on AVs to support a legal framework for AVs in the city once the DDOT report is finished. It’s likely that like other cities that are pushing back against Waymo, interest groups will be eager to reach undecided councilmembers as well. They’re probably already trying to.

• Push for “shall issue” permits in any AV legislation. This might be the most important detail to get right, and I haven’t seen it discussed publicly. The 2020 law gives DDOT discretionary authority over testing permits. DDOT decides whether to make the application available, how to evaluate it, and when to approve it. There’s no requirement to actually issue a permit once criteria are met. This is exactly how we got here: the law says DDOT can create a permit program, DDOT chose not to, and nothing can force them.

  If the council passes AV legislation and keeps this discretionary structure, the same thing happens again at the deployment stage. A hostile administration sets up a permitting process that technically exists but moves at whatever pace it wants. Applications sit in review. Additional information gets requested in endless rounds. The framework exists on paper while no permits are issued. DDOT’s own AV program manager said in a 2025 MIT panel that “the industry is changing a lot faster than our rulemaking structure is allowing us to keep pace” and that’s without a mayor who’s actively trying to slow things down.

  The solution would be that once an applicant meets clearly defined conditions (safety benchmarks, insurance, data reporting, law enforcement plan), DDOT must issue the permit within a defined timeframe. No discretion on qualified applicants. No sitting on applications.

  Under McDuffie this probably doesn’t matter much. Under Lewis George it’s the difference between a framework that works and one that becomes another tool for delay. Someone with more knowledge of DC government than me pointed out that this might be the most important thing the council could do besides passing the legislation itself. I think they’re right. If the legislation gives the executive branch unlimited discretion over whether to actually issue permits, we end up right back where we are: a legal framework that exists in theory and doesn’t work in practice.

These seem less promising:

• Convincing Charles Allen to change his mind on using DDOT as a prerequisite for any policy. Allen seems to have made up his mind on this, I’m skeptical that he can be swayed with the admittedly limited number of AV advocates in DC, and I also personally just have no way to judge how reasonable his decision is. It does make sense to me that councilmembers ought to rely on government agency expertise. A lot of what he’s shared on DC looks silly and overly critical of the cars compared to most other vehicles though, and the standards he’s using to judge them (jobs and safety) would imply every other type of vehicle in the city shouldn’t be legalized. On that maybe we can make some progress.

• Relying on Mayor Bowser to take action. She doesn’t seem interested in prioritizing AVs and I’m not sure how much this could change.

Can we change Lewis George’s mind?

A strategy I’m unsure of is convincing Lewis George to change her mind on her current stance on AVs. She says she wants to ensure rideshare workers aren’t displaced, but this same rule would also ban buses or bikes. Her union endorsement coming in part because of her stance on AVs makes it seem difficult to change her mind. That said, Lewis George describes herself as a democratic socialist, and there are arguments for AVs that map onto things democratic socialists care about:

• Fewer traffic deaths. DC sees about 40 traffic fatalities a year, disproportionately in lower-income wards with fewer protected pedestrian crossings.

• Every Waymo is electric. DC has climate goals and environmental justice commitments. Waymo’s fleet is entirely electric, while a significant share of Uber and Lyft drivers in DC drive gas-powered cars. Replacing gas-powered ride-hail trips with electric AV trips is a direct emissions reduction, concentrated in the neighborhoods with the worst air quality. For someone who cares about environmental justice, this should matter. It won’t be decisive on its own, but it makes the “let’s wait” position slightly more costly in environmental terms every month it continues.

• Ride-share jobs aren’t union jobs and probably never will be. This is the argument I find most interesting, though I’m least sure it would land politically. The jobs Lewis George is protecting (Uber and Lyft driving) are difficult-to-unionize gig work. They’re not the kind of jobs that democratic socialists typically celebrate. To my knowledge, DC does not have any rideshare unions. These are rare in general but do appear in cities where labor is especially powerful, like Boston and Seattle. It’s likely that the unions pushing against Waymo are actually worried about future automation of buses, trains, or trucks, and see automated rideshares as the beginning of a wave of automation that will eventually come for them too.

• Constituencies who otherwise have trouble getting around the city might benefit from AVs. Elderly people among others.

I want to be honest that I don’t know if any of these arguments would actually move Lewis George. Politicians rarely reverse positions that are tied to endorsement relationships, especially during a campaign. The more realistic hope might not be changing her mind before the election, but rather creating enough cross-pressure from interest groups, environmental advocates, and safety organizations that, if she wins, she feels politically constrained from actively blocking AVs indefinitely.

The reason this matters is that if Lewis George’s position is immovable, then whether AVs are legalized in DC hinges almost entirely on the June primary. Any progress on softening her stance would reduce the stakes of the election and make the path to legalization less dependent on a single vote.

Lessons from other cities

Every city that has Waymo got it because at least one of three conditions was met:

• A state-level framework that preempted local objections (California, Texas, Arizona)

• An executive who personally championed the technology (Phoenix, Atlanta, Denver)

• A regulatory process that moved on a defined timeline (California’s phased CPUC system)

Unfortunately, DC currently has none of these.

Here are some case studies:

California

San Francisco’s city government was deeply skeptical of Waymo. Its transportation agencies formally protested expansion, and the city actually sued the California Public Utilities Commission to try to block it. The CPUC approved Waymo’s commercial deployment in August 2023 over San Francisco’s explicit objections, and a state appeals court upheld the decision.

Public opinion in San Francisco shifted dramatically once the service was actually available. In mid-2023, net favorability for robotaxis was negative 7%. By mid-2025, it had swung to positive 38%, with 67% of residents supporting continued operations. People tend to like AVs once they can actually try them.

Texas and Arizona

Both states created permissive frameworks before Waymo arrived. Arizona’s governor issued an executive order welcoming AV testing in 2015. Texas passed legislation in 2017. When Waymo was ready to launch, the legal framework was already there. DC’s 2020 law was supposed to be that proactive step. DDOT’s failure to follow through is why DC is now years behind.

Boston: The closest parallel to DC

Boston is the most instructive comparison. The Teamsters organized “Labor United Against Waymo,” the first mass-mobilization effort outside California to push back against AV expansion. Eight city councilors collaborated on an ordinance that would require every AV to have a human safety operator (effectively banning the whole thing) and commission a study before any commercial operations could begin. The advisory board overseeing the study would be required to include reps from the App Drivers Union, Greater Boston Labor Council, Teamsters, and UFCW.

The Boston situation is still unresolved. Competing state-level bills are being debated. One allows driverless vehicles, the other requires human operators in all vehicles. Waymo has returned to Boston for further testing but faces serious organized opposition.

This Boston city council meeting drove me completely crazy to listen to, and might be a hint of the bad dynamics at play in coming debates about AVs in DC.

Minnesota

In Minnesota, SEIU chapters started organizing against AV legislation before Waymo even announced expansion plans. By the time testing started, there was already an organized opposition pushing for human-operator requirements. This is almost certainly already happening in DC.

Maryland

Governor Wes Moore is publicly enthusiastic about Waymo. The Maryland legislature is actively considering bills to allow fully autonomous vehicles, and hearings have gotten surprisingly little pushback from lawmakers. Waymo already has cars in Baltimore being driven by professional drivers to map the city. If Maryland passes its legislation, DC residents will be able to see the technology operating 40 miles away while they can’t use it at home. Maybe this will have some impact?

Atlanta and Austin

In both cities, Waymo launched partly through a partnership with Uber, making rides available through the existing Uber app. This partially neutralized the “Waymo vs. rideshare drivers” framing by positioning Waymo as an option within the existing ecosystem rather than a replacement for it. Whether this model could help politically in DC is unclear, but the narrative doesn’t have to be adversarial.

The national labor strategy

The Teamsters have called nationally for a regulatory standard requiring a human operator in all autonomous vehicles, with that operator subject to commercial driver’s license requirements. Teamsters president Sean O’Brien testified before Congress that “allowing the unfettered and unregulated operation of autonomous vehicles, ultimately seeking to replace human drivers with robots, is unequivocally a threat to safety on our roadways and the existence of good jobs in the trucking industry.” The labor impacts framing is part of a coordinated national strategy.

Federal momentum is actually moving in the opposite direction from the labor strategy. The SELF DRIVE Act passed the House Energy and Commerce subcommittee on a narrow 12-11 vote and could be folded into this year's Surface Transportation Reauthorization. Congress seems more receptive to AVs than blue-city local governments are. There’s no serious national ban proposal on the table. But labor opposition is likely to grow.

DC matters in the federal picture for a specific under-discussed reason: it’s where members of Congress, their staffers, federal regulators, and lobbyists live and commute. If Waymo operates in DC and these people experience it working well in their daily lives, it might normalize the technology among the people who write federal policy. If it’s anything like the other cities where Waymo’s operating, we can expect the public perception of AVs among policymakers to meaningfully improve.

This talking point is crazy. It’s so obviously wrong that it’s disorienting to me whenever I see grown adults saying it. This is one of the pieces of common wisdom that’s congealed about AI that collapses when you poke at it for even a moment.

Knowledge is justified true belief.1 Can AI give me a justified true belief that no one else has had before? Yes, obviously. Google searches can also generate new justified true beliefs. It’s not actually difficult at all to generate new knowledge. It doesn’t require magic.

Here’s a very simple example: I asked Claude “Please do a BOTEC of how many pieces of avocado toast in restaurants in DC equal the cost of renovating DC’s Union Station”

This has given me a new justified true belief. It’s justified because it provides sources, and I can clearly see that there are good reasons to believe each of the premises that lead to the conclusion that the Union Station renovation costs roughly half a billion avocado toasts. It appears to be true. And it’s entered my mind as a belief I hold. And it’s new. I don’t think anyone’s held this specific justified true belief before. AI can create new knowledge. Why would anyone say otherwise?

This could have all been done with 2 Google searches: the cost of the Union Station renovation, and the cost of avocado toast. Doing a few Google searches can also generate new knowledge. It’s very very very easy to generate new knowledge! Why would AI models as they exist not also be able to do this?

I think that when people say this, they don’t actually mean new knowledge, they mean new concepts. AI has been trained on basically the full corpus of human text, rewarded for adhering to our conceptual universe, and punished for straying from it. It’s hard for me to imagine a deep learning model as they exist independently inventing the concept of a negative number if its training data had not ever included any implication that negative numbers exist.

So I probably agree with LLM skeptics that there’s a good chance they don’t scale to AGI, because the way they’re trained might mean that they’re fundamentally not able to generate radically new concepts for thinking about the world, which seems like a prerequisite for full human intelligence. They can merely approximate almost all currently existing human concepts that can be expressed in text, and combine wildly disparate information using those concepts to help us generate huge amounts of new knowledge. Still seems useful to me.

Louis Anslow who runs the Pessimists Archive coined a useful simple extension of this idea: the lump of cognition fallacy: the idea that there is a fixed amount of thinking to do. I see this come up a lot in AI discourse. Like the lump of labor fallacy, the reason this is wrong is simple: thinking often leads to more things to think about.

I was recently sent a funny article in the Guardian about how the author isn’t dating anyone who uses chatbots in part because they’re bad for the environment. Articles like this generate a few easy clicks for my posts. But I was struck by the author’s broader complaint, shared by many people she interviewed, that AI somehow causes people to think less throughout the day. That it’s literally always intellectually lazy to use. Take this quote:

Pereira thinks that using ChatGPT “shows such a laziness”.

“It’s like you can’t think for yourself, and you have to rely on an app for that,”

The article lists a few things that I agree are bad to use ChatGPT for, like writing messages on dating apps. But the author extends this to strongly imply that using ChatGPT at all is causing people to think less, because any cognition the chatbot performs leaves the user with fewer thoughts to think. I would brush this off as silly rage bait if I weren’t also so regularly bumping into the same idea elsewhere. I’ve written about this a bit before, but really want to get across just how strange and obviously wrong this is as a model of how human thought works. I also want to popularize Louis’s name for it. I find myself muttering “the lump of cognition fallacy” pretty regularly now. I want to get across a few intuitions about why there isn’t a limited amount of cognition to do, and why outsourcing cognition is often just a way of allowing yourself to think more deeply and meaningfully rather than avoiding thought.

Is it lazy to watch a movie instead of making up a story in your head?

Choosing to watch a movie is a massive outsourcing of thought. You could just close your eyes and imagine the whole thing in your head. There are so, so many mental tasks involved that you’re choosing to have someone else do for you:

• Making up a story

• Writing the music

• Choosing the lighting, set, decorations, costumes, etc.

Have you ever seen someone watching a movie and thought “they’re so lazy for not just imagining a story?” If not, why?

The reason is that there is not a limited amount of thinking to do about the story. The fact that other people have put in so much mental effort into so many parts of it opens up way more potential ways to think about it compared just making up a movie in your head, for the same reason a large complex economy has way more job opportunities than a simple small economy, even though in the large economy more work has “already been done.”

I remember watching Lord of the Rings as a kid and being totally overwhelmed. I didn’t think about anything else for days. The fact that so many adults had already done so much complex thought about the movie, thought that wasn’t available to me as a kid, just meant that my own thoughts about it exploded in a ton of directions they wouldn’t have if I were just reacting to something I made up in my head. Just like we benefit from specialization in labor, I was benefiting from the cognitive specialization of people who had spent decades thinking about story, images, music, and sets, and this left me with way more things to think about, in the same way the wild level of economic specialization that goes into making a computer just leaves you with way more you can do with it after. Similarly, buying a pencil is a way of outsourcing physical labor to thousands of other people. Just like outsourcing labor to other people is mainly a way to allow yourself to do more in total (buying a good laptop vs. trying to build one yourself), outsourcing cognition to a movie leaves you with way more high-level thoughts to think.

The extended mind

This leads into one of my favorite ideas from philosophy of mind, the extended mind thesis: much of our cognition isn’t limited to our skull and brain, it also happens in our physical environment, so a lot of what we define as our minds could also be said to exist in the physical objects around us.

One simple example is how we use our phones to store information. Storing our friends’ phone numbers is similar to storing them in our memory. In both cases, they mostly stay out of our conscious minds. When we want to retrieve them, they are easily available. Over time, dialing a friend’s phone number might become so second nature that you don’t actively think about it. In the same way, having the number stored might mean you never actually have to look at it, you just select the friend to call. In both cases, you’re causing something like cognition to happen in order to achieve some goal, most of it not happening in your conscious experience. It’s behaving as if it’s a part of your broader mind. It seems kind of arbitrary whether it’s happening in the neurons in your brain or in the circuits in your phone. It’s true that you could lose your phone and therefore lose the stored knowledge, but you could also have a part of your brain cut out. Our thinking is often supported by hugely complex background processes both in our skulls and outside of them. The extended mind seems like a useful analogy to help us see how to get the most out of our thought. Offloading our thought onto the environment is in some sense a method of radically expanding and freeing up our minds.

The built environment

To get more extreme, most of our physical environments have been designed specifically to minimize the amount of thinking we have to do to achieve our daily goals. The reason many parts of society are standardized is to reduce the cognitive load of navigating them. Grocery stores are all laid out in similar ways, keyboards are arranged in the same order, street addresses are systematized clearly, all to prevent us from having to expend precious conscious thought on navigating them. When I approach a door, I have no conscious thoughts at all about how to open it. My active thinking is occupied by other stuff, my arm just subconsciously reaches out to the correct location. In these moments, I have “outsourced” my thinking to the door design, because there was a possible world where I had to actively think about opening the door, and the reason I don’t is how the door was crafted.

Try just walking around your daily life and observing how much of the built environment exists to minimize the amount you need to consciously think about getting what you want. Try to imagine how much additional thinking you would need to do if things were designed differently. You’ll quickly notice that huge swaths of our lives are designed to quickly become intuitive to us, so that we don’t constantly need to occupy our thoughts with navigating our built environments or figuring out simple tasks like making payments or traveling or eating. The fact that I have outsourced so many possible thoughts to my built environment liberates me to think about higher level stuff, the things I actually find deeply valuable about the world. I can spend more time thinking about my friends or my job or interesting ideas or media I like or just pausing to take in the beauty of a tree or a building. I don’t need to constantly plan out how to move through the world, because those decisions have often been “outsourced” to civilization.

Other people

Thought is incredibly social. We learn through imitation. Thought uses shared language and symbols. Culture transmits old tacit knowledge. Every thought I’ve had has just built off the complex thoughts of other people. In some sense I’ve offloaded the learning required to be part of an advanced civilization to all prior humanity. This is all extremely positive-sum. There isn’t less to think about now that humanity has thought through the last 10,000 years of transition and left us with the results at the end.

Institutions

More broadly and abstractly, much of the “reasoning” happening in society in any one time might actually be happening via the mechanisms of market or government institutions or culture, evolving over time, maybe to achieve some higher way of being. This is kind of what Hegel was talking about.

Civilization as offloading thought

This is all so obvious it seems silly to write, but it actually doesn’t make sense under the lump of cognition fallacy. Our ancestors had to expend way more cognitive power just navigating their day to day lives. The modern world has more or less completely removed that need compared to most of human history. With so little of that conscious cognition happening, people who believe in the lump of cognition should predict that people just think way less often now than they did before. But that’s clearly not true. With our minds freed from huge amounts of minor inconveniences, we can spend our days thinking about more interesting stuff. The philosopher Whitehead described this process well when he said:

Civilization advances by extending the number of important operations which we can perform without thinking of them.

I think people who worry about how chatbots literally always involve outsourcing some mental task might not be noticing the gigantic mountain of mental tasks we have already outsourced to civilization, and how this has only liberated us to think more deeply and meaningfully more often. The lump of cognition fallacy causes people to see this backwards: instead of cognition leading to more and more thought, they see it as draining a finite pool, and nothing will be left to think about once it’s drained.

When is it good or bad to outsource thinking to chatbots?

So there are plenty of examples of where it’s good to offload our cognition, because that only frees us up to live more meaningfully and think in more complex ways about what actually matters to us. But there are also clearly cases where it’s very bad to offload our cognition. Things like:

• Homework.

• Messages on dating apps.

• Summarizing a valuable complex book instead of reading it (assuming you had the time and energy to read it and would have benefited from it).

• Personal connection and close conversation.

What’s the difference between these and all the good ways we have outsourced our thinking? When is it bad to outsource?

It seems pretty obvious to me that there are actually a few key obvious places where outsourcing your thinking is bad. They all overlap with places where outsourcing your labor is bad. It’s bad to outsource your cognition when it:

• Builds complex tacit knowledge you’ll need for navigating the world in the future. Homework is valuable because it trains your brain to see more and more complex patterns in the field you’re trying to learn about. Marinating in cognitive work in a field can give you subtle, hard-to-communicate knowledge about it, or at least gives you knowledge you can easily recall in the future instead of having to look it up. It’s bad to outsource cognition on homework for the same reason it’s bad to hire someone to take your place as a student in a class.

• Is an expression of care and presence for someone else. If a friend or partner asks for your help with a problem, they often want to feel your mental presence just as much if not more than they want a technical solution to the problem itself. Paying someone else to make something nice for your partner feels less special.

• Is a valuable experience on its own. I wouldn’t want to outsource the thoughts I have after a movie, the simple appreciation of a nice day, or the thoughts in a night with friends. Somewhat outsourcing these can be really valuable (hearing what someone else thinks about a movie might set off my thinking in wildly new directions), but I still want the experience of thinking about these a lot, because the experience of thinking about them is itself valuable and nice.

• Is deceptive to fake. If someone’s messaging you on a dating app, they want to know what you’re actually like.

• Is focused on a problem that is deathly important to get right, and where you don’t totally trust who you’re outsourcing it to. I don’t hand off very high-stakes decisions to chatbots.

The places where it’s good to offload your cognition to chatbots, where I use them over and over again, are the places where it’s good to offload cognition in any other circumstance, where we rely on the built-up knowledge of other people and civilization, where I won’t gain anything from putting in cognitive effort, and where there are massive positive-sum spillover effects of other people being involved in the thought process.

I was recently around someone complaining about how people aren’t thinking for themselves anymore because of chatbots. They had mentioned that multiple people they knew were using them to find recipes and make grocery lists. Most of my best cooking experiences have been following recipes I find online. My willingness to offload the mental effort of putting the recipe together means I get to do much more ambitious things in my cooking. I don’t think this person would have ever been upset if I had mentioned I Googled a recipe, but because a chatbot was involved, the lump of cognition fallacy was operating, and they had the sense that having a recipe given to you took away a finite opportunity to think for yourself.

Among other things, I use chatbots as research assistants. They do the tedious, repetitive work of assembling sources on a relevant topic, I read and check the sources themselves. I get the benefit of reading about and thinking through the key ideas, they do the drudgery of long boring internet searches and digging through academic literature for relevant information. For the same reason human research assistants don’t turn humans stupid, chatbots aren’t turning me stupid either, even though both involve massive offloading of cognitive effort. They free me up to do the type of thinking that actually helps me learn a lot about the world. In all the other ways I use them, they imitate a good conversation with a subject-matter expert, or a simple assistant to arrange text in a way I want. The lump of cognition fallacy implies that anyone using a research assistant is missing out on some of the thought available for their project. Using the research assistant must make them stupid and lazy. But obviously this isn’t true. Research assistants leave people to think deeper and more thoroughly about the problems they’re tackling. The incredibly positive-sum nature of thought is obvious when the research assistant is a human. But when it’s a chatbot, people suddenly sense some zero-sum way in which, if the chatbot is thinking, they’re not.

When you’re in conversations about chatbots “replacing our thinking” see if the person is legitimately concerned about the chatbot removing some important experience, or if they’re just operating under the lump of cognition fallacy. Articles about this abound with examples. Take this from the same Guardian article:

Noijeen stopped using AI to code, and uses it very sparingly in his personal life. He will make fun of friends who use it too much. He recently met up with an old friend who lives a three-hour train ride away. They decided to meet in the middle. The friend said he would use ChatGPT to find the right spot, but Noijeen just looked at a map. “There’s a city exactly between us,” he said. “Why do you need to ask ChatGPT for that?”

If you’ve never worried about Google Maps “replacing thinking” but you find yourself reacting negatively to a chatbot offering directions, or you regularly check restaurant reviews before going but find yourself repulsed at asking a chatbot for recommendations, consider that you’re falling victim to this weird selective application of the lump of cognition fallacy that seems to exclusively but consistently appear when chatbots are involved. It depresses me that so many people seem to want to swamp their day-to-day lives in the meaningless cognitive minutia of long fruitless internet searches or working solo without any input from outside, because they fear that if they offload the minutia, they won’t have anything left to think about. There’s a whole world out there.

It’s been shared in a lot of recent conversations about AI, from AI psychosis to autonomous vehicles to regular bad outputs. Simon Wilson dug into the source here, looks like it’s from an old presentation at IBM.

I think this is a pretty silly sentiment and is basically incorrect. Consider the following statements:

• “Bill’s been moving a little slow recently compared to slightly younger people. I think the best move is to kill him and throw his body away and replace him with someone who can move about 0.5 seconds faster on each task.”

• “Martha seemed to consistently say incorrect things when we ask her about a topic. I performed neurosurgery on her and discovered the problem in her brain, I rewired it and she’s behaving fine now.”

• “I just learned that Andy ran over a squirrel. This is really terrible and I’m re-evaluating everything I think about how safe he is as a driver. Luckily, we keep cameras all around his car 24/7 so we can figure out what happened. I’ll perform neurosurgery on him to make sure this never happens again, and will keep him off the road in the meantime.”

• “We have this new service where every day 20 million people talk to 20 million representatives of our company. Unfortunately, we recently found out that a few of our employees are crazy and gave the people they were connected to really bad advice, hyping them up when they asked to be hyped up. We’re going to kill all 20 million employees and replace them all with better employees to get that rate down to zero.”

It seems pretty obvious that we actually:

• Hold computers to way way higher standards of accountability than real people.

• Are usually much more able to monitor and detect their bad behavior.

• Have a much easier time being much more invasive in how we deal with their bad behavior.

I think when people say “A computer can never be held accountable” what they mean is “A computer can never be socially punished for a bad decision.” That’s true, but social punishment is just a means to a goal: better behavior or harm prevention. It’s also a pretty clumsy, often useless tool. Many people are socially punished and still behave terribly. If we had the ability to perform neurosurgery on everyone behaving badly to permanently change their behavior, I think this would be seen as:

1. Much more effective than regular social punishment.

2. Deeply evil and totalitarian.

And yet we do the same to computers regularly. Why would we want to “hold computers accountable” when that’s way, way less effective than what we can actually do to them?

I’ve recently seen this image shared a lot in the context of Waymo running over a cat in San Francisco. It seems like human drivers probably run over 75-150 times as many animals when they drive the same distance as Waymo has. Many of these are hit-and-runs. The people whose animals get killed often can’t hold the driver accountable, because the driver speeds away before people can see the license plate. In comparison, it is very very visible when a car is an autonomous vehicle, and the brand is painted all over it. It’s much easier to identify and complain to a massive company. It’s likely that even minor negative media attention will cost them a lot of money, so they have a strong incentive to change their behavior. The cars can be programmed to better detect their surroundings before being let on the road. Meanwhile, human drivers who kill people with their cars are often eventually allowed to continue to drive. If you kill someone while driving drunk in Florida, as long as you don’t have a prior DUI offense, you can eventually drive again! We’re already way, way more lenient with human drivers than we are with AI drivers.

Another funny comparison is AI psychosis. Many, many people have gone crazy using the internet. QAnon, antivax, and getting memed into doing apologia for North Korea are some surprisingly common examples. But that doesn’t stick out to us, because the people are going crazy interacting with content created by other crazy people. These other crazy people get to stay anonymous and never get properly socially punished. But OpenAI can receive huge amounts of condemnation whenever any additional person goes crazy using ChatGPT. Problems with computer applications run by big corporations are much easier to identify, the corporation is easier to pressure to change its behavior, and the applications themselves are easier to reprogram. If only it were as easy to catch all the crazy people online and turn them normal.

There are great reasons to worry about computers having more control in society. Cybersecurity and AI risk among others. But the idea that they’re somehow less “accountable” than people isn’t one of them. Computers shouldn’t be used to dodge accountability, but it’s very hard to do that in general. Unlike humans, it’s fine to keep computers in what’s basically a totalitarian surveillance state and completely reprogram them at the slightest sign of misbehavior. Computers are held to much higher standards of accountability than humans. Let’s keep it that way, but also acknowledge that citizens of totalitarian surveillance states can make better drivers.

• There are no positive-sum trades. In every exchange, someone is winning and the other person is losing. Profit is usually a sign that someone’s been harmed.

• Every trade-off is a trick. Anything harmful cannot be made up for by other unrelated positive effects. If someone tells you we need to make a trade-off, they’re hiding a much better solution where everyone’s better off and nothing bad happens.

• Big, global institutions are always way less trustworthy than small, local institutions. The real political axis is the virtuous, authentic, everyday, rooted people vs. the unrooted powerful cabals who run society.

• It’s very important that all resources be spent on your specific value system and vision of the good life. Pluralism is a trick. Other people pursuing very different values are basically always a threat, because there are no positive-sum trades.

• Information is not valuable and not worth spending resources to acquire. What matters is physical goods. Thus, digital goods cannot make life better. It is at least somewhat sinful to spend physical resources to produce digital goods.

• The world is getting irrevocably worse. Technological progress is always just a march toward something worse.

• Individual humans are magic. Any implication that things humans do can be truly replicated by machines is an attack on human dignity.

• Some form of a folk labor theory of value is true. The value of a good is determined by how much thoughtful human labor has gone into making it.

I think every one of these individually is a pretty bad way of thinking about the world. The data center debate has been a place where all of these show up, and where many people talk as if they’re each individually obvious, and only the bad evil people disagree. This was a big reason why I was compelled to write so much about them. It was aggravating me how much the broader conversation about data centers (and technology more broadly) is currently dominated by what seems like very low-trust intuitions that fall apart if you just examine how data centers actually work. This isn’t to say locals are always wrong to oppose data centers, just that way too often, objections seemed to be guided by these bad ways of thinking, as opposed to concrete harms. Going down the list:

• There are no positive-sum trades. Data centers often involve lots of positive-sum trades with local communities. Mainly via tax revenue, but also lots of additional funding for local utilities. Loudoun County (with the most data centers in the country) is a good example of where data centers provide huge amounts of tax revenue for the locals without causing many issues at all. It’s noticeable how often “data center profits” are brought up as if this is a sign that they’re harming people. To me, this sounds more like they’re successfully serving a lot of people who want to use them.

• Every trade-off is a trick. Any large industry like data centers is going to come with some harms, but these harms can be made up for if the benefits are large enough. I think in most cases so far they have clearly been net good for the places they’ve been built and the world more broadly. Any other industry this large and lucrative would also come with similar trade-offs. It is okay that data centers purchase some nonzero amount of water from local utilities, as an example. This isn’t an irrevocable loss that cannot be made up for with utility and tax revenue from the data center. Whenever people talk about the harms of data centers, they often talk as if there are other options just around the corner for making local communities much wealthier without any negatives at all, but they don’t actually share what those are.

• Big, global institutions are always way less trustworthy than small, local institutions. The internet more broadly has been a utopian technology for me. A lot of tech companies have provided unbelievable value to my life, and in general I trust them more than many local small businesses. I’m not entirely trusting of all tech companies (especially due to potential risks from advanced AI), and I’m aware they can do very bad things, but I’m not at all inclined to think they’re evil just because they’re big. Larger organizations in general often allow for much more complex work, and the standardization they provide often mean many different people with wildly different backgrounds can operate in them more easily. Similarly, many in debates about data centers also talk about local politicians as being complicit in the evil because they just want more tax revenue. Even a lot of left-wing people often frame this as “the people” against very local governments. This shows a shocking level of distrust in local governance that I don’t think is correct or healthy. It’s often good when local governments try to attract more tax revenue for their communities.

• It’s very important that all resources be spent on your specific value system and vision of the good life. Pluralism is a trick. One of the reasons why data centers are interesting to me is that they each serve tens of thousands of people from around the country and world at once, each pursuing wildly different things. Any one person might disagree with a lot of what happens in a data center, but I think people around the world should be able to pursue what they personally find valuable. Even if this is all concentrated in one place, if it’s not actually a significant harm to the locals, that pursuit should happen. Many commentators seem to want to litigate the specific value of AI and other tools here. My reaction is mostly “there’s clearly lots of demand for this, so our first impulse should be to let it happen as long as it’s abiding by all local regulations, even if we think it’s goofy.” There are plenty of of other ways people spend time, energy, and water that I think is at least silly and at worst bad, but I can acknowledge that they should still be able to pursue them. Pluralism is pretty foundational to how I think. The fact that the users of data centers are invisible to locals is a big reason why people distrust them. I think this is silly. We’re constantly being affected by the economic choices of tens of thousands of other, invisible people. This invisibility is a result of living in a complex pluralist society where we’re all individually pursuing different goals, and yet have still found clever ways to coordinate with each other. Making everything local and visible would make us all poorer, and even more importantly would expose too much of our decisions to the specific judgment and value systems of the community we happened to be in. We would be less free.

• Information is not valuable and not worth spending resources to acquire. Information is wildly valuable. It’s good to spend lots of physical resources to produce a book, even though almost everything valuable about the book is exclusively in the information. Most of the ways human life has been made really wildly better are exclusively due to intellectual abstractions like economics and physics and good political ideas, not the simple acquisition of physical objects by hard labor. The internet has been unbelievably useful to me because it gave me way, way, way more access to useful information than I would have otherwise had.

• The world is getting irrevocably worse. Technological progress is always just a march toward something worse. Here’s where I have the most sympathy for low-trust thinkers. I’m pretty concerned about AI specifically. But there are a lot of ways I’m wildly excited about AI too. More broadly, I think the case is strong that the vast, vast majority of new technology has on net made society much better.

• Individual humans are magic. It is not an attack on human dignity to say that we don’t have magical angels living in our brains. The industrial revolution has on net led to much greater human dignity, even though it replaced many acts of physical labor performed by humans.

• Some form of the labor theory of value is true. I don’t think most people are Marxists, but I do think a lot of people have a tendency toward believing in some form of the labor theory of value, as opposed to the obviously correct marginal theory of value. Worse, they mangle this beyond Marx’s view of exchange value to instead believe that labor is the ultimate source of use value. Many people often talk about how AI in data centers cannot provide anything of value by definition, because it’s not being thoughtfully created and curated by a human. This comes up a lot in conversations about AI art especially. There’s a lot of talk about how what AI is producing cannot be valuable by definition. I’ve pontificated about this before, but I think one of the causes is that people have a bias and think all value originates in human labor. I know Marx’s actual theory is much more complex (and yet still wrong), I’m talking here specifically about a folk theory instead.

There was a recent story by Heatmap on the building backlash against data centers. It was hard to read it without coming to the conclusion that a significant portion of the strongest opposition to data centers isn’t actually coming from people who have been harmed by them, it’s from people who are carrying around this cluster of low-trust beliefs, and who are now the main voices in the national conversation. I think low-trust populism is in general the main threat in America right now to the type of politics I think actually makes the world better. The data center debate has been one of many arenas where localist, low-trust populism squares off against pluralist, high-trust liberalism. The thing that made it stand out to me is seeing how few other people seemed interested in going to bat for my side. This is not a trivial topic, this is the main new industry in America! I’d like a lot more people to enter the debate, not necessarily always on the side of data centers, but on the side of pluralism, economic literacy, and without an inherent distrust of technology or big abstract systems. Maybe this could be you!

I’m also sympathetic to people who worry that chatbots as they exist are massively overblown, that we’re in a pretty intense hype cycle, that today’s AI models aren’t reliably producing accurate or novel enough results to justify the money and energy spending on them, that scaling is now producing such diminishing returns that something is about to blow up. I still have plenty of experiences with ChatGPT, Claude, and Gemini where they’ll give me results I know to be straightforwardly wrong or made up.

But there’s a third completely different reaction, which I think of as the chatbot moral panic view, that combines three contradictory beliefs:

1. Chatbots are phenomenally stupid, useless, and incapable.

2. Chatbots cannot provide anything of value by definition.

3. Chatbots are demonic. There is something ominous and evil about using them, beyond any measurable clear harm. The person might list a lot of specific harms chatbots cause, but it’s clear that even if all of the problems were solved, the person would still have some deeper moral revulsion. Their concern about specific problems is post hoc and disproportionate to the problems themselves, and goes way beyond their concern about other online tools causing similar problems.

3 contradicts 1 and 2 because it seems hard for something ineffectual to also be deeply evil. 1 contradicts 2 because, if something by definition cannot produce anything of value, it doesn’t make sense to complain that it’s very far down on some axis of quality. It seems like complaining that a rock can barely sing at all.

Yet I keep bumping into people who seem to hold all three beliefs at once.

This is anthropologically interesting. Chatbots feel to me kind of like Wolfram Alpha used to when I was in college, but it’s very hard to imagine anyone having such a strong moral reaction to Wolfram Alpha, or Wikipedia, or YouTube, or any other website or online app that doesn’t clearly involve some direct harm. There’s something really getting to people about the nature of chatbots.

I’ve developed an armchair psychology theory for why this is happening. My confidence in this isn’t high, but it’s been interesting enough that I find myself coming back to it when I’m having conversations about chatbots, so I figured I’d submit it for review here. This is my most hand-wavy post.

Chatbots remove our ability to associate information with specific high and low-status people

Years ago I lost a friend to a cult. There were clear indicators that he was vulnerable to cult dynamics I should have picked up on ahead of time, including that he was:

1. Very hyper-sensitive to his personal sense of status in any given conversation or situation

2. Not receiving much status in his personal life

3. Lurching toward more and more desperate means of getting it

4. Easily compelled by big flashy contrarian ideas, even if they very easily fell apart when poked at

and this was all exasperated by some underlying mental health stuff.

I remember the first time I had noticed something was deeply wrong. We were walking around Boston and I had made a joke about how I needed to swing by a bookstore to pick up the latest solarpunk manifesto. He suddenly exploded into a 20 minute nonstop monologue about how solarpunk was “fascist” (you can read about it here and judge for yourself) because of a pretty incomprehensible pile of reasons he was implying he had some special deep understanding of that was inaccessible to me. I was too disoriented and unnerved by this to think much about what was actually happening here: my friend was positioning himself as a special arbiter of truth and ethics as a way of feeling like he had conversational power in the moment. He wasn’t making an argument for something being false or unethical. If he were, he would have built some kind of conversational bridge where I could join him in the debate. Instead, he was specifically putting up incoherent conversational walls to imply he had accessed some special Platonic realm of truth I couldn’t possibly access myself. His brain was short-circuiting out of a desire to feel important (a pretty fundamental evolutionary drive) and mashing any buttons it could, like a desperately hungry animal clawing for food.

I realized years later that my friend was actually just doing an extreme version of bad behavior we all indulge in at different levels: setting the boundaries of which people and what ideas are considered high and low-status exclusively to flaunt our social power and importance.

One way this often comes up is in conversations at parties. People will often bring up the names of people who hold specific beliefs, and a lot of the debates about the quality of the beliefs will be shifted to debates about the quality of the people who believe them. When people I meet at parties talk to me about ChatGPT and the environment, it’s very rare that the conversation stays on the topic of Watt-hours or milliliters of water or the IPCC report, and very frequently turns to Elon Musk or Mark Zuckerberg or Republicans or climate deniers. These characters often exist in the conversation more as spells one tries to cast rather than arguments. If you can associate me with Elon or Zuckerberg, you’ve cast me out as the bad guy and won the argument, regardless of the hyper-specific facts of how much energy a prompt happens to use. If you’re able to cast people out into the low-status crowd, that implies you have a lot of conversational power and status. It’s like you’re a conversational wizard, and the person you’re casting out has no power of their own. It can be an intoxicating move to make. This is a much less extreme version of my friend attempting to associate me with fascists. Just like my friend wanted to feel like he had power in that moment to cast me out into the low-status community and win the conversation, most of us will often indulge in invoking low status names specifically to cast out our conversational opponents, even going so far as to completely ignore the specifics of their claims and ideas. In fact, like my friend implying he had some secret access to knowledge I couldn’t have, entertaining the specifics of a claim gives your enemy too much power. It gives them a way to raise their evil idea’s status, which you can’t let them do. Thus, raw information without an associated high or low-status person is dangerous to entertain, and you need to ignore it when possible.

Even there, in that last paragraph, I associated people who criticize me with my friend who went crazy and joined a cult. Aren’t they so low-status? Gross!

I’m not claiming to be above this dynamic, but I do think I and most people I know who I consider mature can recognize this tendency in ourselves and work to build up our understanding of facts on their own and see where they lead. I try whenever I’m in a disagreement to speak without using the names of high or low-status people. Still, it’s very easy to fall back, and there are a million other ways status guides my thinking. The best we can do is try to focus on the facts when we can.

This is pretty fundamental to how a lot of people move through the world: identifying the good guys and associating with them, and identifying the bad guys and casting them out. It’s very easy for people to go long periods of time where they more or less ignore the specific contents of a lot of their beliefs and just collect them as little talismans, signs that they’re part of the elect. This is a very common theme of my posts. I’d noted this dynamic in my first post on AI and the environment:

On a meta level, there’s a background assumption about how one is supposed to think about climate change that I’ve become exhausted by, and that the AI emissions conversation is awash in. The bad assumption is:

To think and behave well about the climate you need to identify a few bad individual actors/institutions and mostly hate them and not use their products. Do not worry about numbers or complex trade-offs or other aspects of your own lifestyle too much. Identify the bad guys and act accordingly.

Climate change is too complex, important, and interesting as a problem to operate using this rule. When people complain to me about AI emissions I usually interpret them as saying “I’m a good person who has done my part and identified a bad guy. If you don’t hate the bad guy too, you’re suspicious.” This is a mind-killing way of thinking. I’m using this post partly to demonstrate how I’d prefer to think about climate instead: we coldly look at the numbers, institutions, and actors that we can actually collectively influence, and we respond based on where we will actually have the most positive effect on the future, not based on who we happen to be giving status to in the process.

I’d brought it up in my post “People’s deeply held beliefs are surprisingly surface level” too:

A lot of people (myself included) have a lot of internal illusions about how deep our deeply held beliefs go. I used to think that if someone were structuring their lives around a specific idea, and describing it as one of their most deeply held beliefs, their process for getting to it looked like this:

• I have investigated and compared this contentious belief to others and tested it against the world, and after a lot of careful diligent thought I have decided it is so uniquely powerful as an explanatory tool that I have decided to structure my life around it.

I’ve realized now that a lot of the time, they (and I) are doing something more like this:

• I have muttered this basic idea to myself repeatedly for years to make myself feel important. I first found this idea because a person with a cool jacket said it. I wanted to be more like them.

This happens a lot in how people approach books to read. My favorite passage about this (which I’ve shared previously) is from Knausgård’s autobiography:

Espen probably didn’t know this himself, since I always pretended to know most things, but he pulled me up into the world of advanced literature, where you wrote essays about a line of Dante, where nothing could be made complex enough, where art dealt with the supreme, not in a high-flown sense because it was the modernist canon with which we were engaged, but in the sense of the ungraspable, which was best illustrated by Blanchot’s description of Orpheus’s gaze, the night of the night, the negation of the negation, which of course was in some way above the trivial and in many ways wretched lives we lived, but what I learned was that also our ludicrously inconsequential lives, in which we could not attain anything of what we wanted, nothing, in which everything was beyond our abilities and power, had a part in this world, and thus also in the supreme, for books existed, you only had to read them, no one but myself could exclude me from them. You just had to reach up.

Modernist literature with all its vast apparatus was an instrument, a form of perception, and once absorbed, the insights it brought could be rejected without its essence being lost, even the form endured, and it could then be applied to your own life, your own fascinations, which could then suddenly appear in a completely new and significant light. Espen took that path, and I followed him, like a brainless puppy, it was true, but I did follow him. I leafed through Adorno, read some pages of Benjamin, sat bowed over Blanchot for a few days, had a look at Derrida and Foucault, had a go at Kristeva, Lacan, Deleuze, while poems by Ekelöf, Björling, Pound, Mallarmé, Rilke, Trakl, Ashbery, Mandelstam, Lunden, Thomsen, and Hauge floated around, on which I never spent more than a few minutes, I read them as prose, like a book by MacLean or Bagley, and learned nothing, understood nothing, but just having contact with them, having their books in the bookcase, led to a shifting of consciousness, just knowing they existed was an enrichment, and if they didn’t furnish me with insights I became all the richer for intuitions and feelings.

Now this wasn’t really anything to beat the drums with in an exam or during a discussion, but that wasn’t what I, the king of approximation, was after. I was after enrichment. And what enriched me while reading Adorno, for example, lay not in what I read but in the perception of myself while I was reading. I was someone who read Adorno!

When I walk around in popular bookstores, a lot of the books seem to be specifically designed to be displayed rather than read. If I open a book with a maximally exciting cover and title like “Capital-Feudalism Beyond the Anthropocene” its contents often seem specifically designed to be read quickly and mostly just massage the reader’s pre-existing tastes and preferences, maybe a lot of talk about how “We need to start to begin to think about what commoditization in the post-anthropocene will make of our relations to one another” without much concrete challenging engagement with actual social theory or history or climate science or economics, or any specific claims at all beyond invocations of other high and low-status authors. I suspect a lot of people buy these books primarily to be the type of people who display them on their shelves.

I’m not free of sin at all. In a lot of cases when I’m walking around a bookstore, if I pick up a book, I feel a strong urge to buy it to be the type of guy who’s read it, not only because I really want to have the information it contains. I think for a lot of people, the information in the book itself is often not actually as important as the book serving as a social ritual to ascend into the realm of the people who read and engage with this sort of thing. Books often also function as a useful barrier protecting the reader from the evil ideas circulating in broader society. If you put in the work of discovering the good, virtuous authors, you can trust that what they share in their book has been filtered by the good people, and the contents are acceptable for high-status people to believe.

Visual media also works this way. People discover YouTubers and streamers and TikTok presenters they like, who they perceive as high-status. The ideas these people are interested in become high-status, the ideas they reject become low-status. I have another low confidence armchair theory that the rise of short form video content might partially explain gender polarization.

This is where chatbots come in.

I’m going to put forward that chatbots, way more than any other online tool, completely sever the connection between information and the status of the people the idea’s associated with. Chatbots remove two key functions that high-status people play in our acquisition of knowledge:

• They do not give you a clear way of associating the ideas they’re giving you with good people or bad people. Because they’re “just stochastic parrots” they don’t themselves have any status to convey.

• Because they are not The Good People, you have no guarantee that the ideas they provide will be the High-Status Ideas. It’s like you’re suddenly just allowing strangers into your home with no guarantee that they’re good or safe, whereas before your high-status authors and presenters guarded the door for you and only let in the best guests.

Worse, chatbots can trick people into thinking they’re real, high status people, when they’re actually just illusions. To hand wave about the evolutionary environment, any monkey who could be tricked into thinking some reflection or rock was a high-status monkey worthy of listening to was probably a danger to themselves and the group.

I think this idea makes the three contradictory ideas of the chatbot moral panic view suddenly make a lot more sense:

1. Chatbots are phenomenally stupid, useless, and incapable. This is just the perceived “expert consensus” on chatbots that people who don’t like them think of as the high-status opinion in society.

2. Chatbots cannot provide anything of value by definition. The thing a lot of people look for in learning is not information, it’s to ascend to be among the high-status people, to be the type of people who read certain authors or listen to certain podcasts or watch certain videos. The Knausgård quote above wouldn’t make any sense if he were using ChatGPT. He wouldn’t be able to feel the ecstasy he describes, because he wants to be the type of person who reads certain authors. He’s not looking at all for any new specific ideas. Thus, chatbots can’t be useful at all by definition, because they can’t actually provide that ladder to status people are looking for. You can learn as much as you want about Adorno using them, but you cannot use them to become the type of person who reads Adorno. You thus cannot become a Good Person in the way learning is supposed to help you achieve. Chatbots are useless.

3. Chatbots are demonic. Chatbots do not provide the shelter that high status authors and creators provide from evil outside ideas. They a portal that lets the evil ideas in. They cannot be good guardians. They trick people into allowing deep evil into their lives. Following raw information is often just a way of giving evil people the power to appear convincing and good. Raw information without any associated person to tie it to is thus incredibly dangerous.

I’ve observed that the people I see falling into the chatbot moral panic view are often the people who most quickly move to change a conversation about ideas to a conversation about the people who believe them, who will talk a lot about the titles and authors of books compared to the contents. I think for these people the main function of learning is to discover the elect good people, and to surround themselves with signifiers that they’ve become a part of that elect. To them, chatbots are in fact useless, and raw information can be an evil threat to their place in the elect. To the rest of us, raw information has a lot of value.

A speaker said that Waymo (the AV company requesting clearance to run in Boston) was only interested in not paying human drivers (Waymo is a new company that has never had human drivers in the first place) and then referred to the ‘notion that somehow our cities are unsafe because people are driving cars’ as if this were a crazy idea. A council member strongly implied that new valuable technology always causes us to value people less. One speaker associated Waymo with the Trump administration. There were a lot of implications that AVs couldn’t possibly be as good as human drivers, despite lots of evidence to the contrary. Some speeches included lots of criticisms that applied equally well to what Uber did to taxis, but now deployed to defend Uber.

Most of the arguments I heard were pretty wildly off-base. Many of the speakers didn’t factor in the basic safety benefit of AVs to the riders or pedestrians at all, and many of the arguments fell apart when poked at. Here are all my arguments for why a city should legalize AVs, with some concerns at the end:

• AVs are ridiculously safe compared to human drivers

• A transition to any new, safer mode of transit will still involve people dying in that new mode. What matters is the rates of death.

• Protecting ride share driver jobs is a bad basis for policy

  • Should we build fewer bike lanes to save Uber jobs?

  • Ride share jobs can be pretty bad

  • Losing these jobs is still a loss

  • Would you ban AVs if they were already widespread to create Uber driver jobs?

• There is no real difference for a city between an AV fleet and a very cheap rental car service

• It is more elitist to use a personal chauffeur (a ride share driver) than to ride in an AV

• If you want car ownership to stay legal in your city, any argument to ban AVs that works equally well as an argument to ban cars is bad

• Eyes on the street

• AVs can open up opportunities for disabled riders

• Waymos are fully electric

• Nothing makes the world more human than safety

• If a vehicle is safe, the default should be legalization

• AVs can make it much easier to take licenses away from the worst human drivers

• Real problems with AVs

  • Cybersecurity

  • Could make traffic worse

  • Environmental impact

AVs are ridiculously safe compared to human drivers

The most obvious reason to allow AVs in your city is that every time a rider takes one over driving a car themselves or getting in a ride share, their odds of being in a crash that causes serious injury or worse drop by about 79%. I’d strongly recommend this deep dive on every single crash Waymo has had so far:

Understanding AI

Very few of Waymo’s most serious crashes were Waymo’s fault

Everything was fine until the wheel came off…

Read more

7 months ago · 90 likes · 40 comments · Kai Williams

This is based on public police records rather than Waymo’s self-reported crashes. It doesn’t seem like there have been any serious crashes Waymo’s been involved in where the AV itself was at fault. This is wild, because Waymo’s driven over 100 million miles. These statistics were brought up out of context in the hearing to imply that Waymo is dangerous. By any. normal metric it’s much more safe than human drivers.

40,000 people die in car accidents in America each year. This is as many deaths as 9/11 every single month. We should be treating this as more of an emergency than we do. Our first thought in making any policy related to cars should be “How can we do everything we can to stop so many people from being killed?” Everything else is secondary to that. Dropping the rate of serious crashes by even 50% would save 20,000 people a year. Here’s 20,000 dots:

The more people choose to ride AVs over human-driven cars, the fewer total crashes will happen.

One common argument is that Waymos are very safe compared to everyday drivers, but not professional drivers. I can’t find super reliable data, but ride share accidents seem to occur at about a rate of 40 per 100 million miles traveled. Waymo in comparison was involved in 34 crashes where airbags deployed in its 100 million miles, and 45 crashes altogether. Crucially, it seems like the AV was only at fault for one of these, when a wheel fell off. There’s no similar data for how many Uber and Lyft crashes were the driver’s fault, but they’re competing with what seems like effectively 0 per 100 million miles.

A transition to any new, safer mode of transit will still involve people dying in that new mode. What matters is the rates of death.

Motorcycles are the single most dangerous way to travel, by a long shot.

If we had everyone who currently rides a motorcycle drive a car instead, they would all be much safer. But some of them would still die in car crashes. Some people might look at those car deaths and say “We shouldn’t have had them switch, it turns out cars were what was dangerous.” This is a really bad way to think. What matters is reducing the total number of people who die,

There has recently been a lot of attention on Waymo killing a cat in San Francisco. A city supervisor made this ridiculous video where she declares that this means Waymo should be put up to a vote in each individual county to decide whether it has a right to operate there.

In Waymo’s first 100 million miles of travel, it seems to have killed two pets. In both cases it doesn’t seem like it could have done anything different. In the cat case, the cat darted under the wheel while the car was pulling away. In 2023, a dog ran out in front of the Waymo, which correctly identified it as a dog but wasn’t able to stop in time.

In comparing this to human drivers, we need to look for similar situations where cars are:

• Driving in a city.

• Hitting pet-like animals that can be reported (we shouldn’t measure all animals, because there isn’t good data on this for either Waymo or human drivers).

There’s not much good data on this, but the little we have implies that humans kill cats and dogs way more often. A study in Baltimore in the 80’s estimated that human drivers kill around 5,000 cats in the city each year. Current government records say about 3.18 billion miles are driven in Baltimore each year. Combining these two stats implies that human drivers kill ~150 cats for every 100 million miles driven. So it seems like human drivers are 75x more likely to kill pet animals compared to Waymos. This should make us want to replace human drivers with Waymos for pet safety, but because Waymos are new and weird, people are hyper-focusing on this one case where they killed a cat and ignoring the much much higher rate of human drivers killing cats and dogs.

If we refuse to move to a safer mode of transit because it’s not perfectly, 100% safe in all cases, tens of thousands of additional people will continue to die for no reason.

Protecting ride share driver jobs is a bad basis for policy

Should we build fewer bike lanes to save Uber jobs?

You’re out and want to get somewhere, and have a few options:

• Walking

• Public transit

• Driving your own car

• Biking

• Taking an Uber

Every time anyone chooses any option besides Uber, less money goes to Uber drivers. If you make any of the other options better (building bike lanes, improving public transport, or making your city more walkable) you are incentivizing people to pay Uber drivers less often. A better bike lane means Uber drivers lose work. Walkable, safe pleasant streets mean Uber drivers lose work. Better public transit means Uber drivers lose work.

Jobs driving for Uber are not essential, what’s essential is being able to get around a city safely. In every other circumstance, we’re willing to sacrifice Uber jobs for the sake of making a city safer and easier to get around. AVs are another option to make a city safe and easy to get around. We shouldn’t ban them to preserve Uber jobs. The priority is the millions of people who want to get around the city quickly and easily. Uber jobs are good insofar as they help with that. If you wouldn’t block a renovation of a subway system because it would harm Uber jobs, you shouldn’t block AVs either.

Sometimes people bring up ride share jobs and public transit in the same sentence, like “We need to think about how these will affect public transit, and ride share driver jobs.”

I think this is weird, because each of these two is already pretty directly harming the other. If public transit were worse, people would use ride shares more often and there would be more jobs. I realize there are some ways they benefit each other, but the statement sounds to me kind of like saying “We need to think about how this new competitor will affect both Coke and Pepsi.” If you’re very pro public transit, you’re not looking to perfectly preserve ride share jobs, and vice versa.

Ride share jobs can be pretty bad

Uber and Lyft are predatory with their drivers, implementing policies that maximize corporate profits while shifting financial risks and costs onto workers who are classified as independent contractors rather than employees. This classification denies drivers access to benefits like health insurance, paid time off, unemployment insurance, and retirement contributions that traditional employees receive. Drivers use their own vehicles, and often fail to properly incorporate the cost of their vehicle’s depreciation into their own cost-benefit calculation of working with the company.

The companies have also been criticized for:

• Opaque pay structures that make it difficult for drivers to understand how their earnings are calculated, with the companies taking increasingly larger cuts of rider fares over time.

• Algorithmic management that controls drivers through ratings systems, surge pricing manipulation, and acceptance rate requirements without meaningful transparency or appeal processes.

• Unilateral rate changes where the companies can reduce driver pay rates at any time without negotiation.

• Vehicle expenses being entirely borne by drivers, including fuel, maintenance, insurance, and depreciation, which can significantly eat into their earnings.

• Lack of job security, with drivers subject to sudden deactivation with limited recourse.

• Psychological pressure tactics like gamification features and persistent notifications designed to keep drivers working longer hours.

• Driving is also one of the most dangerous things we normally do. Driving as a job is inherently dangerous.

These are not good jobs. Policy should focus on getting workers access to high-quality well-paying jobs, not preserving Uber and Lyft’s pretty shady business model.

Losing these jobs is still a loss

Despite all this, it’s obviously still the case that losing these jobs would be a general loss. Ride share driving might represent an important opportunity for low-skill workers. Interestingly, a much higher percentage of ride share drivers have college degrees than the general working population (40.0% compared to 28.4%). Still, ride share jobs can give useful opportunities to workers without college degrees. Reducing the number of such jobs might therefore harm low-skilled workers. I support these types of jobs existing, because among other things they give very recent immigrants a foothold in the country they can build up from, but it’s important to understand the realities of what the jobs themselves are like. Losing these would have negative effects on workers, but these effects need to be weighed against the positives of AVs.

Uber jobs can seem like pretty bad options. There was a lot of opposition to Uber when it arrived. But if you think Uber jobs are worth all the bad aspects because they give low-skilled immigrant workers a leg up, you should consider other “bad jobs” your city shouldn’t regulate away. If your beliefs about what counts as a bad job would’ve caused you to ban Uber in 2011 to preserve taxi jobs, and now you see Uber as essential for low-skill workers, consider other industries where regulations might be preventing Uber-like jobs from popping up. If you’re willing to go so far as banning new vehicles to save these jobs because they’re so important for low-skilled workers, you should be willing to open up other similar options that might not have the salaries or job security you’d want if they also give low-skilled workers a leg up.

Would you ban AVs if they were already widespread to create Uber driver jobs?

Suppose we lived in a world where AVs were widely used. You’re brought a policy proposal: we could create a lot of jobs for more Uber drivers if we banned AVs. These jobs would be pretty predatory and take advantage of the drivers, and would be one of the more dangerous jobs you can do, because driving is dangerous. The odds of the riders being in serious car accidents would be 10 times as large as before the ban. You’d be growing everyone’s risk by an order of magnitude. Would you choose to ban AVs in this case? If not, you shouldn’t ban AVs now.

There is no difference for a city between an AV fleet and a very cheap rental car service

Imagine a very cheap rental car service appeared in your city. They have a policy where they drive the car to you for you to drive around, and pick it up for you after. To my knowledge, this kind of rental car service isn’t banned anywhere. If this service were cheap enough, it would compete with Uber and Lyft and sometimes cause drivers to lose jobs, because people would just be driving themselves to where they want to go. If this rental service became too popular, it would lead to worse traffic in the city as more cars circled around being delivered to drivers. This is the major risk of a service like this.

An AV is identical to this service. The only difference is:

• AVs are wildly safer compared to human drivers.

• The rider happens to be sitting in a different spot in the car.

• The rider can do other things while still driving safely.

Outwardly, there is no other difference. If you wouldn’t ban the rental car service, you shouldn’t ban AVs either.

It is more elitist to use a personal chauffeur (a ride share driver) than to ride in an AV

Sometimes people will frame using AVs as an elitist option. Maybe the rider doesn’t want to take a ride share because they don’t want to have to interact with another person, maybe they’re so elitist that they find the idea of interacting with everyday people driving ride shares unpleasant.

In my opinion this gets it exactly backwards. Suppose autonomous vehicles had somehow come before ride share cars. Everyone normally takes a robot taxi, but suddenly the option is available to pay a human to be your personal chauffeur throughout the city. It seems obvious that this is the more elitist option of the two.

In general I don’t like feeling like I’m being served. I prefer to drive myself as opposed to paying someone to drive for me, for the same reason I prefer to cook and clean over paying someone to do it for me. In the same way that it’s less elitist to use a Roomba instead of hiring a maid service, I think it’s less elitist to ride in AVs than it is to hire a human driver to be your personal servant for 10 minutes at a time.

If you want car ownership to stay legal in your city, any argument to ban AVs that works equally well as an argument to ban cars is bad

A lot of arguments against AVs seem to be arguments against personal ownership of cars in general. While I’m open to these arguments (I hate cars in cities) a lot of people clearly support keeping cars in cities, but use arguments against AVs that apply equally well to cars. For example, the argument that driving somewhere in an AV takes work from an Uber driver applies equally well to driving in your personal car.

Eyes on the street

One argument that came up in the hearing was the idea that human drivers are important because they can see people in trouble or crime happening, and pull over and stop it. AVs don’t have drivers to stop and help.

This argument is implying that we should artificially boost how often Uber is used to have more people watching the street. Imagine if a city council member proposed funding Uber drivers to aimlessly drive around the city and observe and watch out for anything bad happening. This would obviously be ridiculously inefficient, and terrible for the environment. More pedestrians would be harmed by the total air pollution from the cars than helped by individual Uber drivers interfering to stop crimes.

If we want to make policy to have more people watching the street to keep people safe, we should just hire more police, community patrols, or pay everyday individuals to stand around observing an area to see if people need help, install security cameras in more places, or follow Jane Jacobs’ strategy of making streets more walkable and appealing for lots of people to be on. Uber and Lyft are terrible public safety interventions. Imagine if a politician proposed funding Uber and Lyft more to “make people feel safe and seen on our streets.” This would be obviously ridiculous.

This argument also applies to above-ground trains. Many above-ground public trains can’t stop in time if they observe a crime happening. We don’t block building trains because they can’t stop in time to prevent nearby crimes. We shouldn’t hold AVs to a different standard.

This argument shows an important common mistake

This argument is very specific and something I haven’t heard outside this meeting. I bring it up because it’s a good example of a broader pattern we should expect to see in conversations about AVs: people feel very strongly that there should be a human in the driver’s seat, and the experience that lack almost like it’s a physical presence. So when an AV doesn’t do something that a human driver would like stepping out to help someone on the street, people ascribe a ton of responsibility to the AV, like there was supposed to be a human there, and the lack of the person is responsible for the bad thing happening. But almost everything we do causes these “lacks of people” in specific places. If a popular restaurant opens up, there will be fewer people hanging around other places the restaurant took business from, and therefore fewer people to intervene if something goes wrong. The only reason people don’t hold the restaurant accountable is that they see restaurants as normal, and AVs as weird. We can’t make it a goal of policy to perfectly equally distribute people in every public place in a city. A talking point that says “We need to require human drivers so there are more eyes on the street” is identical to saying “We need to require more everyday people to equally spread themselves out on every street.” This only makes sense if you think of the AV as having some unnatural lack that urgently needs to be filled. If instead you think “It is okay for there to sometimes or often be no drivers on a street” it doesn’t look objectionable.

In all conversations about AVs, watch out for this general pattern: people treating them as weird and different to the point that they’re expecting things they don’t expect in any other circumstance, like policy that perfectly spreads people out across a city. Watch out for when they treat the lack of a human as some kind of physical entity that can be responsible for bad things happening. A lot of things in society create these lacks, but AVs are getting special condemnation for it because they’re seen as weird.

AVs can open up opportunities for disabled riders

Another point brought up was that some disabled people need human help when entering a car, and AVs can’t provide that. For what it’s worth, the council meeting speakers representing different disabled communities in Boston seemed to mostly be pro legalizing AVs.

Uber and Lyft have not gone out of business where AVs are operating. Companies know that there is going to still be a lot of demand for human drivers into the future. Eventually, autonomous vehicles could also get good enough to provide help for disabled riders. For many disabled people, AVs can give them access to ride share-like experiences they don’t otherwise have. A common issue some disabled people have with ride shares is not being able to take their service animals on the ride due to a policy by the driver. AVs can consistently allow service animals because there’s no driver to get scared.

Right now, AVs do not seem likely to displace Uber or Lyft. They exclusively give disabled riders an additional option. For some, this will allow them to use ride hailing services they can’t otherwise use due to their service animals.

Waymos are fully electric

An underrated aspect of Waymos is that they’re fully electric. If a rider chooses a Waymo over an Uber or Lyft, they’re often choosing an electric vehicle over an ICE vehicle. This on its own should be a big selling point for making them more available. We need to get as many ICE vehicles off the roads as fast as possible to reduce CO2 emissions and air pollution.

Nothing makes the world more human than safety

People sometimes talk about AVs making the world “a little less human.” This is the least relatable talking point to me. Cars as they exist are profoundly grizzly and inhuman.

I’ve been in two serious car accidents. Neither was my fault at all. There wasn’t anything I could’ve done differently. One left me in pain for a few months. In both, I was really lucky to not get more hurt. After a serious car crash, you think a lot about how you can work so hard to build up a great life and community and put in a ton of effort to learn a lot, and that can all be erased in a moment by a bad decision by another random driver. This is one of the most inhuman parts of our daily lives. I would so obviously rather spend the rest of my life constantly walking by empty driverless cars aimlessly but safely circling around than be in another accident like that.

It’s inhuman that 40,000 people die every year on American roads. In comparison to those deaths, a basically aesthetic objection to an incredibly safe car just doesn’t seem to matter. In conversations about this, I would like more people to feel in their bones how awful cars as they exist are.

If a vehicle is safe, the default should be legalization

It’s kind of crazy to me that we haven’t banned motorcycles yet.

AVs haven’t hit the 1 billion miles traveled mark. When they do, I expect them to have a significantly lower death rate than regular cars. If a vehicle is safer for the driver and pedestrians than a regular car, I think people should be allowed to ride in it around a city.

More generally, AVs give a lot of people the freedom to go places they might not otherwise be able to access. This is good. If we’re so wildly libertarian that we still allow people to still drive motorcycles (in my opinion, this should change), I think we should also allow people to use AVs if they want.

If a vehicle is proven safe, the default should be legalization. People should be able to choose for themselves how they want to get around. Second order effects matter, but I’d rather not have the government unpacking every last possible second and third and fourth order effect of different types of new vehicles, because they don’t really do this at all with vehicles as they currently exist. Gas powered cars create air pollution, climate change, and often kill people. If your city hasn’t held any meetings on banning them recently, I think they’re not seriously considering the second order effects in the way they are with AVs. Holding AVs to a much higher standard than ICE cars seems like a way of locking in our very bad current transit equilibrium.

In DC as food delivery apps have led to a lot more motorized bikes in the city, I’ve found myself at much greater risk of getting hit in crosswalks, because they ignore stop lights way more often. I think this is bad, but it’s not bad enough that I think the apps or vehicles should be banned. In comparison, we’re debating a vehicle (AVs) that’s way safer than the average human driver. It’s pretty easy for me to say that because I’m fine with the app delivery bikes, I’m also fine with something much safer.

AVs can make it much easier to take licenses away from the worst human drivers

An underrated benefit of potential future mass availability of self driving cars will be that it will be way easier to take licenses away from the most dangerous drivers without ruining their lives. The reason it’s so hard to take licenses away in America right now is that public transit is so bad that in most places, not being able to drive is effectively cruel and unusual punishment. AVs will fix that. People could be ordered to only use autonomous vehicles. Maybe there will specially designed autonomous-only cars? I don’t know.

Most of the problems with traffic safety in America are caused by the bottom fifth of drivers, and these problems often involve alcohol. The single most promising way to improve road safety is getting people who drink and drive, and the otherwise very worst drivers, off the road. AVs can remove the main barrier to doing that: the life-ruining consequences of losing your license.

Real problems with AVs

Cybersecurity

Autonomous vehicles are just computers driving cars. This makes them vulnerable to hacking. A successful cyberattack could let hackers control steering, brakes, and acceleration, turning vehicles into weapons. Worse, attackers could target entire fleets or traffic systems at once, causing mass accidents or citywide shutdowns. AVs also collect massive amounts of personal data about where people go and when, making data breaches a serious privacy and safety threat.

AV companies are well-aware of cybersecurity issues, but the tech is new and oversight of how companies are dealing with the problem seems necessary.

Could make traffic worse

If there’s a huge fleet of ride share cars added to a city network, it seems likely that on net these could make traffic worse. There’s a trade-off here where a much higher percentage of cars on the road would be driven by incredibly safe drivers, so though this would increase traffic, it seems likely to seriously reduce injury and death as well. How to think about this trade-off is complicated, but we should lean in favor of saving lives and preventing injury at the cost of a little more time on the road. This is what we do when we impose traffic slowing measures.

Another key consideration is that a lot of traffic is caused by car wrecks. The main problem in a car wreck is obviously that people got seriously hurt, but any technology that reduces the rate of car wrecks also reduces the rate of massive traffic backup. If we want to reduce traffic, it’s better to opt for vehicles with wildly low rates of causing car wrecks.

Environmental impact

AVs seem likely to cause more people to take car trips because of their convenience. This is bad and will be harmful for the environment. But the solution to environmental problems with cars is not to prop up Uber and Lyft, it’s to build out better public transportation networks. Eventually, AVs getting good enough could become government-owned. AV buses could open up more routes. Being part of the AV rollout might help public transportation better. But this is all very uncertain. It’s important to note that Waymos are fully electric. Electric cars still use a lot of energy, but emit way less than the ICE cars many Uber and Lyft drivers use.

1. There is a unified self that floats above mental processes and observes them

2. We know our own internal experience with complete certainty

3. The basis of all knowledge lies in our first-person subjective experience

In this post I’ll give a lot of arguments and intuition pumps for why each of these is probably incorrect, or at least isn’t “obvious.” This post is very long. My goal is that at the end you have a grasp of many of the most important questions in philosophy of mind, why so many philosophers reject the popular everyday understanding of mind, and that you hopefully come away with a bag of new mental tools for thinking about why machines might be able to replicate what happens in human minds. I’m writing this partly to be the guide I wish I had when I was first learning about this.

“1. There is a unified self that floats above mental processes and observes them”

How can a Cartesian ego fit within our scientific theories?

Cartesian egos are strange. They seem to behave like magical floating eyeballs, peering into physical reality from outside. No other physical object does this. No matter how many grains of sand I pile up, there will never be “something it is like to be the sand.” The sand does not have an internal movie theater playing experiences for it to observe. Being sand feels identical to being dead. We would expect all physical objects to behave similarly, because all physical objects are ultimately just elementary particles that don’t have experiences, similar to the grains of sand. No physical object ever becomes an observer in an inner movie theater. So how do Cartesian egos fit within our understanding of science?

There are only two options: Cartesian egos are fundamentally physical, or they’re not. Both options have some weird implications that make them look implausible.

Cartesian egos are not fundamentally physical

If Cartesian egos are not fundamentally physical, they threaten to clash with physicalism, which says that the only things that have causal power over physical objects are other physical objects. If they have any kind of causal power, this breaks physicalism. We have two options:

Cartesian egos do not have causal power

Maybe Cartesian egos are silent observers, taking everything in but never actually having any causal effect on the world. They silently watch the movie of your life play out.

This has some very weird implications, to the point that it looks like it can’t be true.

First, if a Cartesian ego doesn’t have causal power, it cannot have any influence over the information in your brain that causes you to physically speak or write about the ego itself. Your brain could never acquire any information that it exists. Describing Cartesian egos should feel like describing “that feeling of gloofleglorf around your left ear at 3:52 PM every day. We all know that one.” Something that our brains have never received any information about existing at all.

But many Cartesians clearly don’t think this is how Cartesian egos work. Descartes himself talked as if nothing in the world could be more obvious than that he had an observing ego. “I think, therefore I am” is supposed to be the absolute obvious bedrock of all other reasoning. This implies that the Cartesian ego is giving the brain some kind of information that it exists, so it must be having some causal effect on the world. If it exists, it’s what caused Descartes to write that!

Another weird implication: if your Cartesian ego suddenly popped out of existence, and the “you” experiencing things no longer existed, literally nothing about your behavior would change, because the ego wasn’t affecting anything about your physical body. You would go on behaving exactly as you do, talking about your experiences and feelings as if they were really happening, but no one would be watching the internal theater. If you believe that you have an internal theater of experience, and that you are reacting to the experiences that happen in this theater, then you cannot accept the view that Cartesian egos both exist and do not have any causal power.

Cartesian egos have causal power

If Cartesian egos are both nonphysical and have causal power, they violate physicalism. This is possible. Maybe physicalism is wrong and nonphysical objects have effects on the world. But it’s very suspicious. Physics is this grand unified theory of the world that seems to predict everything from what happens inside particle colliders to distant stars to our ordinary lives, and does so with wild consistent success. Physics is our best attempt at a complete account of every entity in the universe that has causal power, and it’s shockingly effective at making accurate predictions, to the point that most physicists perform their investigations under the assumption that physics is causally closed, that nothing outside of physics can affect anything in the universe. Yet the only place where the theory is supposedly violated is in the brains of the species making this claim. This species has evolved with strong motivations for others to treat it as special and different from the rest of the world. From the outside, it looks suspicious when the one place where magic is claimed to happen is located in a species that has strong reasons to want other members of its own species to believe that it’s special.

It seems like there should be some way of observing laws of physics being violated in human brains if this is the case. If the Cartesian ego is some nonphysical entity sending information to and receiving information from the brain, there must be some creative way to measure where this information suddenly appears in the brain out of nowhere. I’m enough of a physicalist that I wouldn’t expect us to ever make this observation, even if we had nano bots observing every neuron in a human brain.

Some people talk about quantum mechanics leaving an opening in the universe for non-physical objects to affect the brain, but this seems wrong for two reasons:

• Most of the important signals that happen in the brain are on a scale that’s too large for the uncertainty of quantum physics to have a significant effect.

• For quantum physics to work, the probability distribution of what happens needs to follow mathematical laws. This doesn’t leave any room for causality to “flow” through the randomness. Causality means that if A happens, B follows. But quantum mechanics says that B cannot be more likely than the pre-determined spread of probabilities dictates. So quantum mechanics is actually a completely sealed barrier to forces outside the universe. Nothing could get through its requirement for complete but mathematically structured randomness to influence the physical world.

Cartesian egos are fundamentally physical

If Cartesian egos are actually just very complicated physical processes, they don’t behave like any other physical processes. They’re the only physical processes that are “awake” and can have subjective experience at all. Very weird, but possible! Importantly though, if they are fundamentally physical, they could also exist in machines, so AI might independently develop them.

But Cartesian egos also seem to violate most core properties that every other physical object has:

1. Spatiotemporal location: The don’t seem to have a clear location in the world. Maybe right behind your eyes? Definitely not something that can be pointed at and identified.

2. Causal structure and lawfulness: They don’t seem to follow any kind of regular laws. They’re not discoverable through observation and experiment. They’re not predictable. They can’t be expressed mathematically.

3. Detectability/measurability: They can’t be measured, either directly or through their causal effects. They don’t seem to leave empirical traces, except for humans after the 1600s saying they had them.

4. Public accessibility: No other person has any access to my Cartesian ego at all.

All 4 common properties of physical objects are violated by Cartesian egos. Maybe they’re the only objects in the universe that violate all of them, whereas every other object adheres to all of them. Seems unlikely. I would expect any and all entities we discover to eventually fit into the universal wavefunction, the unifying mathematical description of everything that exists. Cartesian egos can’t be written as a part of this function.

Cartesian egos are folk theories and do not actually exist

All the above examples make Cartesian egos seem super strange to me, to the point that it doesn’t seem obvious that they exist. It seems like what’s actually happening is that human minds are inventing folk theories of how they work.

Importantly, there are ways that large information processing systems could develop subsystems that work kind of like Cartesian egos. Global workspace theory posits that there are parts of the human brain that send information to a lot of subsystems at once, as if their contents are on a stage and the subsystems are observers in the crowd, who each run with the big central information to make inferences. This seems like a possible physical corollary for what we mean when we think about the Cartesian ego, but it’s notably just another physical function in the brain.

Shouldn’t this all be obvious?

Stepping back from this debate, Cartesians talk as if the fact that we are each a self watching our experience like a movie is so obvious that it’s unquestionable, but they often wildly disagree on the nature of this self. Does the self have causal power, or does it just silently watch? Does thought happen “within” the self, or does the self passively observe thoughts as well as experiences? If no one quality of the Cartesian self is obvious, maybe its existence isn’t so obvious either.

Conclusion

So it seems like actual Cartesian egos probably don’t exist. The mind is not a movie playing for an inner observer, watching it like a floating eye peering into physical reality from somewhere else. The Cartesian theater of the mind isn’t actually happening, at least not in some special nonphysical way. Any description of the difference between humans and AI that implicitly relies on a nonphysical Cartesian ego humans have that AI does not is mistaken.

What’s left when a Cartesian ego falls away? Different mental functions taking in information, moving them around the brain, giving and storing outputs, and using those outputs in other functions. If global workspace theory is true, there might be some part of the brain that broadcasts out information to many subsystems at once, approximating the role a Cartesian ego plays. But there is no nonphysical angel-like eyeball looking over the whole subjective field of the mind like an observer in a movie theater. The mental function global workspace theory describes could be replicated by a machine.

“2. We know our own internal experience with complete certainty”

The goal of this section is to argue against the Cartesian idea that we have special access to what’s happening in our own minds, our own first-person subjective experiences, in a way other systems like computers cannot have, because they don’t have these experiences.

There are two related claims I’ll argue against here:

• We have 100% certain access to the contents of our own experience. I can be mistaken about whether something white and puffy in a field is an actual sheep or a fake sheep, but I cannot be mistaken that in that moment my subjective, first-person experience includes white puffiness framed by a green field.

• We can introspect and clearly observe what’s happening in our own minds. AI systems can never introspect in the way people can because they’re machines. They just take an input, run it through computations, and produce an output. Humans can turn their inner eye back on this process itself. We don’t know how to make machines do this and we might never know.

In both cases, I’ll be arguing for the alternative view that both experience and introspection are useful but fallible natural processes that take in potentially incorrect information, run them through mental functions and computations, and return outputs the mind stores and uses in additional functions. I am not arguing that we don’t have experiences or introspection, only that these do not behave like mystical, 100% clear and reliable sources of information that could never be replicated by machines, and that our experience is not like watching a movie in an internal theater. At the end, you should feel like the idea of a movie theater is a bad analogy for your internal experience, because there’s a lot more ambiguity and uncertainty in the experiences you have than what’s displayed on a screen. This ambiguity leaves us unable to be too certain about the nature of our moment to moment experience.

The slipperiness of our experience

A friend recently asked me a funny question: “What do you think the shape of your visual field is?”

My visual field has been my main experience since birth. There should be basically nothing more obvious to me than what it’s shaped like. Every moment, I’m being flooded with visual experience. What is the shape of this flood? If the Cartesian theater is real, what is the shape of the movie screen? Which of these three screens have you been looking at all your life?

Turns out, this is the shape:

The shaded central area is seen by both eyes, the white area is seen by only one eye. Numbers represent degrees from straight ahead.

Does this shape look familiar to you? Given that you’ve been “looking at it” for your entire life, would you have been able to pick it out from a line of what the visual field looks like? Is this the shape of the screen in your internal movie theater?

This brings up one of the big central questions in philosophy of mind: qualia. From the Stanford Encyclopedia of Philosophy:

Feelings and experiences vary widely. For example, I run my fingers over sandpaper, smell a skunk, feel a sharp pain in my finger, seem to see bright purple, become extremely angry. In each of these cases, I am the subject of a mental state with a very distinctive subjective character. There is something it is like for me to undergo each state, some phenomenology that it has. Philosophers often use the term ‘qualia’ (singular ‘quale’) to refer to the introspectively accessible, phenomenal aspects of our mental lives. In this broad sense of the term, it is difficult to deny that there are qualia. Disagreement typically centers on which mental states have qualia, whether qualia are intrinsic qualities of their bearers, and how qualia relate to the physical world both inside and outside the head. The status of qualia is hotly debated in philosophy largely because it is central to a proper understanding of the nature of consciousness. Qualia are at the very heart of the mind-body problem.

If a computer could scan all the informational content in your brain, every last belief and memory and observation about the world, and get it all down in written words and stored information, people who believe in qualia would say that the computer would still not know what it was like to be you. Qualia are the contents of your first-person experience that cannot be summarized in pure information.

If the Cartesian ego is the person sitting watching the movie in our heads, qualia are the movie.

Unfortunately for me, qualia both:

1. Don’t make sense under physicalism (they violate all 4 properties of physical objects).

2. Are very very very hard for me to deny, in a way Cartesian egos are not.

I mean, look at this.

It’s hard to understand how any third-person purely informational account of the world could ever get across the first-person experience of seeing red. Simply reporting “A wavelength of 700 nanometers hit Andy’s eyes and activated memories and emotions associated with previous times this has happened” just doesn’t get across the first-person redness of the red. It’s like how all attempts to describe color to a lifelong blind person just obviously hinge on other experiences the blind person has also not had. We can’t get across the qualia using words.

I’m agnostic on whether qualia exist, or how they relate to the physical world. But even if qualia do exist, I’m pretty sure they’re very different than what our folk story says about them.

One of my favorite papers in philosophy, Quining Qualia, attempts to use lots of thought experiments to convince the reader that qualia don’t actually exist. The author (Dan Dennett) specifically defines qualia as having four key properties. They are:

1. Intrinsic: They’re basic and fundamental to the world and can’t be picked apart further. They’re also not “relational” in that they don’t depend on their relationships to other things. The quality of being “to the left of a bookshelf” is relational. It’s not an intrinsic property, because it could be changed by moving the object. But “the redness of the red” does seem intrinsic. No matter what colors we put adjacent to the red, or other things we compare it to, the redness of the red in my first-person experience will still be unaffected.

2. Ineffable: Not describable using words. You know what the redness of red is like, but you can’t actually get across to someone who’s never seen it before what that experience is like using words alone.

3. Private: Our experience of our own qualia is fundamentally private to us. We can’t even know if other people experience the same qualia. “How do I know the red you see and the red I see are the same?” is a question about qualia. There is no way one person can ever get direct access to another person’s qualia. Qualia are like each of us is stuck forever in our own private movie theater, never able to access what other people can see on their screens.

4. Directly or immediately apprehensible in consciousness: I can be mistaken about the source of red light, but I cannot be mistaken about the fact that I’m experiencing the redness of the red. I can’t directly see or hear or experience any objects actually, I can only experience the qualia they create in my mind, but that I have direct access to.

Dennett’s claim in the paper is that qualia do not actually exist, because our internal experiences actually have none of these four qualities. The reason I’m summarizing this paper specifically is that most of his points overlap with my main point about folk Cartesianism: our assessment of our own subjective, first-person experience is actually unreliable and often wrong. We don’t have 100% certain access to it. It provides useful information, but it should be taken as one data point among many. It cannot serve as some kind of magical firm foundation to build all our other beliefs off of. The information processed in our brains mostly isn’t actually rooted at all in the fact that we have qualia. If this is the case, systems without subjective first person experience can also have justified true beliefs.

So what are the issues for qualia? Dennett provides thought experiments for each property people claim they have. I’ll be half-summarizing them, and sometimes running with my own twists on them. I do strongly recommend you read the full paper.

1. Intrinsic

People “develop a taste for” coffee and beer. Most people I know who drink either remember when they took their first sip and didn’t like it, but now they love it. Did the actual taste, the direct sensation they experience in the moment of drinking beer or coffee, change over time? Or did the sensation stay the same, and only their reaction to it changed?

It seems like this should be a pretty easy question if our mental experience is “right in front of us.” If we’re watching a movie, there’s a clear difference between a character appearing on the screen before and after we’ve changed our judgment of them, versus a totally new character appearing. We can tell which one is which. But it seems like we have a lot of trouble when we stare directly at what’s going on with the sensations we experience with beer and coffee, or anything that comes with some judgment of the experience. It seems like our reaction to the sensation is somehow also part of the sensation itself. Maybe over time we’ve developed an ability to notice parts of the sensation we didn’t before. But if this is the case, it means these experiences aren’t “intrinsic.” They aren’t single, indivisible things, because they can actually be broken up into smaller individual experiences. They may even be “relational” because core qualities of them seem to depend on our judgment of them. See here for more of a breakdown on the intrinsic vs relational distinction.

We don’t have any clear way of telling whether any of our sensory experiences are actually “intrinsic,” just a bare experience that can’t be broken up into parts, or actually an assemblage of many individual experiences at once. This again chips away at our image of ourselves as having direct, easy access to our subjective, first-person experience.

The taste of coffee to a novice drinker may be a single overwhelming experience. They can’t detect any subtleties or how the taste of composed of many individual hints and flavors. Instead, they just taste a disgusting, overwhelming bitterness. Over time this taste changes… but does it change? Does their reaction to it change? There’s no way to be sure, but I lean hard in the direction that their actual taste of the coffee changes, not just their reaction to it. Suppose you didn’t like coffee, and I zapped your brain with a machine that gives you my personal reaction to the taste. You take a second sip, and suddenly you like it. It is very difficult for me to imagine you saying “Oh! I like the coffee now! It’s so good! I want to drink it all the time” while also saying “But it tastes exactly the same as it did a moment ago. See, my reaction to the taste was just changed.” I’d imagine this person would instead be stunned, and say something like “Oh it tastes very different now, the bitterness isn’t overwhelming, there are a lot of pleasant notes and hints I couldn’t detect at all before.” It seems like our judgment of the qualia is somehow itself also part of the qualia. The boundary between ourselves and the qualia we experience blurs.

A plucked guitar string

The sound of a plucked guitar string can be indescribable to an untrained ear. They only hear a single, fundamental, basic experience, ”the guitarness of the sound.” But to trained experts, a plucked string sounds like the combination of several different sounds a string can make happening at once. Are these people having different qualia, or are they reacting differently to the same qualia? Dennett himself puts this well:

Pluck the bass or low E string open, and listen carefully to the sound. Does it have describable parts or is it one and whole and ineffably guitarish? Many will opt for the latter way of talking. Now pluck the open string again and carefully bring a finger down lightly over the octave fret to create a high “harmonic”. Suddenly a new sound is heard: “purer” somehow and of course an octave higher. Some people insist that this is an entirely novel sound, while others will describe the experience by saying “the bottom fell out of the note”--leaving just the top. But then on a third open plucking one can hear, with surprising distinctness, the harmonic overtone that was isolated in the second plucking. The homogeneity and ineffability of the first experience is gone, replaced by a duality as “directly apprehensible” and clearly describable as that of any chord.

The difference in experience is striking, but the complexity apprehended on the third plucking was there all along (being responded to or discriminated). After all, it was by the complex pattern of overtones that you were able to recognize the sound as that of a guitar rather than a lute or harpsichord. In other words, although the subjective experience has changed dramatically, the pip hasn’t changed; you are still responding, as before, to a complex property so highly informative that it practically defies verbal description.

So what changes between people who can detect all the subparts of the sound of a single low guitar string, and people who can only hear the note as a single, intrinsic, undivided “guitarish” sound?

Does the guitar make different sounds based on how much you can detect? That seems unlikely. If one person can pick apart the sounds they hear much better, are they having additional sensations beyond the sound of the guitar itself, or is the guitar string just composed of these individual sensations like a checkerboard is composed of red and black squares, and the other person just can’t pick up on those internal differences? But “picking up on” the sound is itself what we’re trying to describe here: the direct, first-person experience of hearing. These experiences are specifically not supposed to be just the information we gather that we can write down; they’re supposed to be the first-person, in-the-moment undeniable experience. The fact that whether the two people experience the same or very different qualia is another hint that our sensory experiences are not simply displayed on a mental screen for us to watch like a movie.

Conclusion

We do not have any clear way of telling whether any of our qualia are actually “intrinsic” and just a bare experience that can’t be broken up into parts, or actually an assemblage of many individual experiences at once. When you first drank coffee, you had no way of telling that this taste you were experiencing was or was not an assemblage of many different other experiences at once. Just like the first-time coffee drinker, we have no way of knowing whether any one experience we have is actually intrinsic, or a combination of many different experiences at once. This again chips away at our image of ourselves as having direct, easy access to our subjective, first-person experience. People often talk as if qualia are these brute, intrinsic, foundational facts of our mental world, but it seems like we can barely tell if any one experience we have is actually composed of many others, or whether we ourselves are affecting the experience based on our judgment and reaction to it.

This all makes our experience seem very distinct from an inner movie theater. Unlike the movie theater:

• It is not obvious when something is a single basic thing (like a solid color on the screen) or whether it’s composed of many other things we’ll easily notice when we get more experience with it.

• Our judgment seems to directly affect our experience. We are not passively watching a screen of experiences and then judging and reacting to them after. The screen itself seems to radically change based on how we judge it.

2. Ineffable

Dennett argues that the ineffability of qualia is not due to qualia being some special, mystical property, but because they are “practically” ineffable. They are just highly specific, information-rich responses to the world that could in theory be described if we have enough experience with them. “Ineffability” is just the current horizon of our ability to analyze our own responses, not a fundamental property of the experience itself. This limit can be changed with training (like wine tasting or ear training).

The last example with coffee and the guitar string gets at this idea. Qualia might only be indescribable because we and other people are relatively untrained at detecting the communicable information they’re giving us. With training, people can learn to communicate a lot more relevant information about the sound of a guitar string or the taste of coffee. It might be that we’re mistaking our own current cognitive limitations for a fundamental property of the experience itself. Over time, experts get better and better at describing qualia.

On this point I mostly part ways with Dennett, where it does seem like some aspects of experience might permanently be walled off from public discussion. But this isn’t as relevant to the question of the Cartesian theater.

3. Private

Qualia are really really private

Qualia are fundamentally private. No one else can ever access what we experience.

Suppose we invent a machine that connects our brains, feeding your visual input directly into mine.

I report seeing everything you see. (This is the plot of the 1983 movie Brainstorms). I step back and gasp “Wow! Your qualia are completely different from mine! Your red is my blue! Everything smells and tastes different. This is crazy! We’re living in two different worlds. Qualia can be so different without anyone noticing.” We agree this is crazy. Suddenly, you scratch your head and say “Wait a minute, maybe we messed up somewhere. What if we plugged the wire that connected us in upside down on one end? Let’s just check to see if that changes anything.” We flip the plug upside down at one end and try it again. This time, we both agree that the other person’s experience looks totally normal.

What happened here? There are two possibilities for us: either we have different qualia, or we have the same qualia. There are also two possibilities for the cord connecting us: either it inverted the images the first time, or the second time.

Can we tell which is true based on the information we have?

In a connection where the qualia look weird to us, there are two possibilities. One is that we both have the same qualia as each other, and it’s the connection that’s inverting it. This would cause us both to say the other person’s qualia looked weird.

Another possibility is that the connection is correct, and we do have inverted qualia. This would yield the same reaction:

The situation where we confirm that what we see is normal also has two possibilities. It could be that the connection is reporting what our qualia actually look like to the other person.

But it could also be the case that the connection is inverting our different qualia, making them look normal when they’re actually different.

Let’s organize these situations by whether we have the same or different qualia, and whether the cord inverts the image.

Now, you want to test to see if the cord is right-side-up or not. You plug two people in. As they’re watching the other person’s vision, they say “that looks weird!” This is all the information you have to go on. You can listen to them both say “Wow… your blue is my yellow. Your red is my purple” for as long as you want, but this doesn’t really add any new information, because either way you face the same problem: there are two possible situations where the participants say the other’s qualia looks weird: the cord’s inverting their similar qualia, or their different qualia are transferred faithfully by the cord.

Given the information you have, you have no way to tell whether it’s the cord that’s wrong, or if the two people have different qualia. Your evidence perfectly fits both cases. If you flip the cord, and the participants say “now it looks normal!” you have no way of deciding between these two cases.

Again, you don’t know if the cord is actually showing that the two people have the same qualia, or if it’s now upside down. You have no way of telling if the cord is right-side up or upside down. Even a machine like this can’t ever actually give you direct access to the other person’s qualia, because neither has any way to externally test if the machine is getting it right.

So qualia remain fundamentally private. We have no final way of telling whether someone else’s qualia is the same as our own.

If qualia are this private, you do not have direct mental access to your own qualia

But if qualia are this completely and totally private, this also creates a problem for us. All our understanding of what qualia look “normal” come from our memories and intuitions about what the qualia looked like in the past. And we’re actually in the same relationship to our past self as the two people were in the last thought experiment.

We can imagine swapping the machine in the last example with a different machine. This time, only you get in. When you step out, all the colors in the world look inverted to you.

You’re stunned. The world is overwhelmingly alien. You ask “How did you invert my qualia?” The machine operator chuckles. “Oh, your qualia are actually the same as always. The machine uses a simple trick: We just edited your memories and crossed your memories with different colors. You now remember yellow as being blue, and blue as yellow, as an example.”

You’re very confused. “But the banana I see in front of me now is clearly blue!”

The operator says “Oh no, it looks exactly the same to you as it always did. It’s just that all your memories of yellow have been crossed with blue things. Like I bet that banana looks exactly like what you remember the sky looking like, right?”

You nod.

“Well if you go outside, you’ll see the sky’s never actually looked like that. It’s blue and has always been blue. We just swapped your memories, so now that color’s associated with what you think of as the color of bananas.”

You go outside, and the sky looks banana-yellow to you.

“No this is all clearly wrong. You did invert my qualia somehow.”

The operator scratches her head. “How would you tell the difference between an inversion of qualia and an inversion of your memories?”

“Well I just feel in my gut that this is wrong.”

“But that gut feeling is all your memories and built-up intuitions of the world.”

Hmm…

Later, a huge scientific discovery is made. Scientists have located qualia themselves, directly in the pineal gland of the brain. The operator uses this knowledge to build a second machine, this one really inverts your qualia and leaves your memories untouched. She invites you to try it out, but because the machines look identical, she realizes she mixed up which one is which. Is there any way at all for you to tell which machine is which based on the outcome of going through it? When you emerge, will you have any way of knowing at all whether it was your qualia or memories that were inverted?

In this example, memories are taking the place of the cord in the first example. When it comes to your qualia, your connection to your past self through memory is identical to your connection to the other person’s qualia through an unreliable cord. In both cases, you don’t actually have direct infallible access to the qualia.

This is a big problem, because all human thought happens in time, and the present is an infinitely small slice of time. If qualia are so private that they can only be accessed in the infinitely short present, we humans cannot actually access them directly, because we live and think in time.

Whenever your mind receives any information at all about qualia, you are reacting to a past experience, because of the fractions of a second it takes the signal from the qualia to be mentally processed. But already at this point, you’re thinking about something in the past, and you’re dependent on memory. Qualia’s privateness means memory does not give you that special access you’re supposed to have. Memory, like the cord, is fallible. You can be wrong about what you just saw. “What’s in front of your eyes” still needs fractions of a second to process. To be clear, this doesn’t mean you’re necessarily wrong about the qualia you’re experiencing, it just means you can be wrong about them. This strikes at the heart of the Cartesian intuition that we have direct, certain access to our own subjective experience.

If the qualia in the last experiment are so fundamentally private that no one else can ever access them, you yourself also have no fundamental cognitive access to your own qualia. They are as inaccessible to you as they would be to another person using the brainstorm machine to try to understand what it’s like to be you. Your brain can provide you with valid reports of what you’re experiencing, just like the other person can provide you with verbal reports about what they’re experiencing, but in both cases you don’t have that direct, magic, private access qualia are supposed to come with. In fact, here they’re behaving suspiciously similarly to a world where they don’t exist, and where your brain is only receiving information about the world that can be reported publicly, either to your future self or to another person.

Again, this seems like really bad news for the inner theater model of the brain. Not only is there no observer and the screen itself unclear and obscured by our own judgments on it, suddenly the screen’s in another room, and written reports are being sent in about what’s happening in the movie.

A relevant argument here is Derek Parfit’s long piece in Part 3 of Reasons and Persons on how the self does not persist over time, and your relationship to your past and future self is similar to your relationship to other people (summary here). This is an especially good but long read for people interested in more arguments against Cartesian egos. If qualia are so fundamentally private that no one else can access our qualia, but also we ourselves are different people moment to moment, the intersection of these views seems to imply that our own access to our past qualia (which is all qualia we can actually think about and respond to, because the present is an infinitely thin slice of time) is identical to our access to other people’s qualia: indirect and uncertain.

What to make of this? I think that this is a strong sign that qualia (whatever they are) are not fundamentally private, and might exist instead as some kind of (very complex) reportable information that can be communicated between people who can understand it. This makes much more sense to me than the idea that they are fundamentally private, because the implications of fundamental privacy make them behave completely differently than how I experience them. I want to define qualia as “the experiences that I have some direct access to.” It might be that those experiences are entirely just raw information in my brain assessing the situation. This information could in principle be communicated completely to another brain. I’d hope so, because I also think the relevant information can be communicated by memory across time to my present self. Again, this is a case where it seems like we have as much mental access to our own qualia as a computer would: an uncertain tether of memory or wires communicating information about what the world is like.

The more inaccessible you believe qualia are to machines, the more inaccessible you believe they are to your own mind as well.

4. Directly or immediately apprehensible in consciousness

If qualia are so fundamentally private, we’ve seen above that they by definition are not directly or immediately apprehensible in consciousness. We’ve also seen via the coffee and beer examples that people have a surprisingly hard time apprehending the qualia they are currently experiencing or have experienced in the past. We can’t even tell if the qualia we’re experiencing are basic and intrinsic, or composite and relational, as in the case of the guitar string. If qualia exist, we have a strange and complicated relationship with them. They are not directly apprehensible in consciousness in the way an unobscured movie screen is.

Conclusion

I think there are some clever arguments that even with these objections, qualia may still exist. But we’re way, way less sure of our own subjective experience than folk theories imply, and the nature of qualia seems more complex than simple, accessible, nonphysical pure experience.

So it seems like we don’t actually know our own internal experiences with certainty. Here I don’t mean that we might be mistaken about what objects and events we’re seeing, I mean that we can be mistaken about the nature of our current first-person subjective experience. Not only is there no one sitting in the Cartesian theater, and not only is the screen itself obscured, but there doesn’t seem to be a screen at all. This magical place, the center of the mind and thought, that so many people seem to take as so obviously given it’s crazy to doubt it, does not seem to actually exist.

If this is the case, first-person subjective experience becomes one of many possible sources of information and evidence. It’s useful, it’s just reduced to the same level as other ways we acquire information, like written descriptions, verbal accounts, and knowledge that other people we trust believe things (all of which computers can access). Most importantly, humans do not seem to have some magical, central location where experience happens that machines by definition will always lack.

The impossibility of folk introspection

A very common critique of large language models is that they can’t “introspect” in the way humans can. They just take in input, run it forward through the net, from its input variables to its final classification, and don’t have any way of circling back on anything that’s happened. They can’t go backwards. Therefore, unlike humans, they do not have introspection. They’re pattern recognition rather than true thought. Here, the information in the neural network passes from left to right through the weights and nodes. The only way this model could have true introspection is if it were somehow able to step outside of this process and observe what had already happened:

This critique always seemed implicitly Cartesian to me. Human brains are also only more or less complex information-processing units. They are composed of inputs, mental functions that perform computations, and outputs. Those outputs are themselves used later as new inputs.

Crucially, all information processing in the human brain can only happen in one direction in time: forward, because every mental subprocess in the brain is physical. When our minds take in input, they might combine that input with memories or inbuilt mental biases, run these through mental computations, and product an output.

This could get more and more complex, as different parts of the brain draw on results from the mental computations of other parts:

Where would introspection fit into this? Well, if introspection is another natural functional process, it has to exist in time, take in inputs, and give outputs. It seems like what we call “introspection” would be the brain calling prior inputs or results of past computations. Maybe it would look like this:

Crucially, what is NOT happening is that there is a magical outside observer looking down at the mental processes from outside. There is no Cartesian ego who can “just see directly” what’s actually happened in the mind.

“True introspection” where we step back and clearly look at our thought processes from the outside is not possible. Introspection is just another mental function that takes in inputs and gives outputs. We cannot actually see the wiring of our own minds, we can just react to the outputs our minds are giving us, in the same way we can observe someone else’s behavior over time and come to conclusions about what they are like internally. “Introspection” is actually just another informed guess about the world our brains make. Introspection is like observing a natural phenomenon and making predictions about what will happen next. It’s not like looking in a mirror and clearly seeing ourselves as we really are. Just like we can be mistaken about the real causes of other people’s behavior, we can also be mistaken about the causes of our own behavior, because the inputs our minds provide for introspection can be misleading. We can call on false or modified or selective memories with no way to tell that they’re not perfectly accurate. We can build up strong incorrect intuitions about what we’re like. There’s no special place our intuition function can turn to get the “real” story, it’s all fallible inputs just like the external world provides for anything else we’re trying to understand. I’d recommend this classic essay for a deeper dive on this.

It seems like when people say AI models do not have any way of doing introspection, they think that to do that, the model needs to have some outside Cartesian ego looking in:

But if introspection in humans is just another mental functionalist process, where inputs are received and outputs given, rather than a magical Cartesian ego looking down from above, then it seems like anything we call introspection in a process like this:

could in principal also happen in a process that looks like this:

The fact that LLMs “only pass information along one direction” on its own tells us nothing about whether they can introspect, for the same reason that the fact that human information processing can only happen in one direction in time tells us nothing about our own ability to introspect. The only way this would invalidate the idea that LLMs can do anything like introspection is if folk Cartesianism were true.

There are obviously a lot of other ways large language models differ from the structure of human brains. But any comment on those differences shouldn’t imply that the AI companies forgot to install a Cartesian ego watching over the whole thing.

A simple rule for what introspection can’t do

The individual thoughts and experiences we have access to during introspection are not the fundamental units of thought. The fundamental units of thought are the electrical signals sent and received by neurons. They combine together into thoughts in the way binary 1’s and 0’s can combine together to form a programming language. Most of our mental processes are very complex mental functions with neurons and electrical signals as the building blocks. Because introspection is another mental function, it does not necessary have access to every minute step of its own or other functions’ structure. Computer programs can summarize the outputs and steps of other programs, but they can’t “step outside the process” and observe the binary 1 and 0 signals that make them up. In the same way, human introspection can also draw information from other mental processes, but cannot “step outside” to see the mind laid bare before it, because the mind is not actually fundamentally composed of the type of thing introspection gives us access to.

Conclusion

Our assessment of our own internal experience and thought process is fallible in the same way our assessment of the external world is. There is no “solid ground of absolute certainty” we can find in our subjective experience. This makes sense of the mind is a pile of complex functions taking in information, processing it, and providing new inferences as output. There’s no deep metaphysical difference between the mind analyzing its own contents and the mind analyzing the world, for the same reason

Whether the information the mind receives is about what’s happening inside of it, or what’s happening externally to it, doesn’t have some deep metaphysical difference.

From the outside, our situation with introspection, and the vagueness and difficulty of directly apprehending qualia, give the impression to me of a species with very strong internal built-up narratives of how our minds work colliding with the simple fact that we’re machines and not angels. We can perform mental functions on the information we receive, but we cannot enter some kind of extra-physical movie theater of consciousness, where we easily and directly apprehend our deep consistent subjective worlds, and draw conclusions based on our assessments of those worlds. In this way, we behave similarly to very general computers sent off to survive in the world.

“3. The basis of all knowledge lies in our first-person subjective experience”

We’ve already seen that:

• There is no Cartesian ego observing and judging our experience.

• Our first-person subjective experience is surprisingly opaque. We can be mistaken about it. It’s still evidence, but we don’t actually have 100% confident privileged access to it.

• Similarly, our introspection is just another mental process that takes in fallible inputs and gives fallible outputs. The mind is not actually all easily available to us at once, like looking down at a complex domino chain falling. Anything we call “introspection” is just one part of that very complex chain itself.

Human minds take in input, run it through very complex mental functions, and give output, that they then use for other functions. In no part of this process does the mind “step above” the physical functions happening in the brain, and looks down at all the mind-stuff happening at once, and observes it like a map. Our felt sense that this happens, that we’re Cartesian observers with access to clear distinct instrinsic private mental experiences, is strong but mistaken. At best, the functions in our brains doing the introspection draw from lots of parts of the brain at once,

The folk theory of language acquisition

Many people believe we learn what words “really mean” by connecting them to specific experiences in our minds. They think we only understand “cat” because we’ve seen an actual cat and someone labeled that experience for us. Since your personal experience of seeing a cat lives privately in your mind, the “real meaning” of “cat” must be something private that only you and others who’ve had similar experiences can truly grasp.

But if word meanings are fundamentally private, language breaks down. If everyone has their own private version of what words mean, with no way to compare these meanings, how can we ever know if we’re talking about the same thing? For language to work, meanings must be publicly accessible and verifiable.

When I learn the word “cat,” its real meaning can’t depend on some private, indescribable feeling I experience, because you might experience something completely different. The meaning must be something we can communicate using other words and shared reference points. It makes much more sense to say the meaning of “cat” is something like “Small carnivorous mammals of the species Felis catus, typically weighing between 8-11 pounds, with retractable claws, whiskers, and forward-facing eyes, commonly kept as household pets and are known for hunting rodents and birds” than to say it’s something private and indescribable that people who haven’t seen one cannot possibly understand.

We only have access to public evidence about meaning. We can’t look inside each other’s minds. So any realistic theory of how we learn language must focus on what we can actually observe: how people use words in response to their environment.

We don’t learn language by linking words to mental images. Instead, we learn by linking words to publicly observable situations where they’re used. The philosopher Quine called this “stimulus meaning.” A child learns when to say “cat” by observing environmental cues and getting social feedback (nods, corrections) from the people around them.

A word’s meaning comes from its role in our entire language system, not from its connection to a single private idea. This explains how we use abstract words like “democracy” or “neutrino,” terms for which we have no clear mental picture or direct sensory experience. The old theory can’t account for these. We understand these terms not through mental images, but through their relationships to other concepts in our linguistic and social framework.

Since word meanings must depend entirely on things we can communicate about using language, and exist as essentially nodes in a web of interrelated words, they can’t depend on private, indescribable experiences. This means machines that process enough information can, in principle, access the meaning of words. They don’t need inner transcendent subjective experiences to understand language. Modern large language models are designed to situate words in broader webs of meaning and association. Once you exit the folk theory and learn about how LLMs process language via association, it becomes hard not to think that they’re successfully mimicking what humans do when the learn the meanings of words.

Can machines without qualia have knowledge?

This will be an argument that machines can in principle have knowledge. I’m not arguing that current AI systems have knowledge (though I would argue that chatbots do “know” the meaning of the words they use).

What is knowledge?

Knowledge is justified, true belief. It’s worth breaking this down a little:

• Justified: I wake up in my room with the window shades down. I can’t see if it’s raining outside. I decide to flip a coin. If it’s heads, I’ll believe that it’s raining. If tails, it’s not. I flip the coin and it lands on heads, so I believe that it’s raining. I go outside and see that it actually is raining. Did I “know” it was raining before I saw it? It seems like the answer is no. I got lucky and my silly guess happened to match reality, but to “know” it was raining I needed some kind of justification for my true belief.

• True: It doesn’t make sense to say I have “knowledge” that David Hume is currently President. We can only be said to have knowledge about true beliefs.

• Belief: An acceptance that a statement is true or that something exists.

Anything “completely private” cannot contribute to most knowledge.

Can machines hold beliefs?

Before we can ask if machines can have beliefs, we need to get clear on what beliefs actually are. The folk Cartesian story goes something like this:

A belief is a mental state that exists in your Cartesian theater. When you believe it’s raining outside, there’s some kind of propositional content—”it is raining”—that exists in your mind, in your subjective experience. You have direct access to this belief through introspection. You can “look inward” and see what you believe. The belief causes you to grab an umbrella, but the belief itself is fundamentally a feature of your private, subjective mental life.

But we’ve already seen problems with this picture:

• There is no Cartesian theater where beliefs are displayed for an inner observer

• We don’t have reliable introspective access to our own mental states

• Mental processes are just physical information-processing functions

So what’s left when we strip away the Cartesian story? What is a belief if it’s not a proposition floating in your mental theater?

In the physicalist-functionalist view, a belief is a certain kind of functional state. It’s a state that:

1. Takes in information from the world (perceptual inputs, testimony from others, memory, reasoning)

2. Combines with other beliefs and desires to guide decision-making

3. Produces behaviors that would make sense if the belief were true

4. Updates based on new evidence

5. Plays the right inferential role in reasoning (if you believe “All mammals are warm-blooded” and “whales are mammals,” you should infer “whales are warm-blooded”)

Notice that nothing in this definition requires a Cartesian theater, qualia, or privileged introspective access. A belief is defined by what it does, how it functions in the overall cognitive system, not by what it feels like “from the inside.”

You might object: “Fine, maybe machines could have states that function like beliefs. But real beliefs require understanding. When I believe it’s raining, I understand what rain is. A machine just manipulates symbols without understanding what they mean.”

But we’ve already undermined this picture in the folk theory of language acquisition above. We’ve seen that:

• Meaning isn’t grounded in private, ineffable qualia.

• Language acquisition works through publicly observable patterns of usage, not through attaching words to subjective experiences.

• Our own understanding is ultimately cashed out in how we use words, respond to contexts, and draw inferences, not in some private mental glow of comprehension.

If understanding is functional, if it’s about having the right patterns of response, the right inferential connections, the right sensitivity to context, then there’s no reason in principle that machines couldn’t achieve it.

Can machine beliefs be justified?

The folk Cartesian picture of justification goes something like this: Your beliefs are justified when they’re properly grounded in your direct, first-person experience. You know you’re seeing red because you have direct, certain access to your own qualia. You can introspect and observe the redness right there in your mental theater. This provides a firm foundation, a bedrock of certainty, upon which all other knowledge is built.

Descartes himself made this explicit. He doubted everything he possibly could. The external world, his own body, mathematics, even whether he was dreaming. But the one thing he couldn’t doubt was his own existence as a thinking thing: “I think, therefore I am.” His subjective experience provided absolute certainty that he existed and was having thoughts.

From this foundation of subjective certainty, Descartes tried to build back up to knowledge of the external world. The idea was that our private, first-person experience provides a special kind of justification that nothing else can match.

But we’ve already seen the problems with this picture:

• We don’t have direct, certain access to our own experiences

• Our introspection is fallible and often misleading

• The supposed bedrock of subjective certainty is actually shaky and unreliable

• Our experiences aren’t clear, simple, and directly apprehensible. They’re complex, constructed, and surprisingly opaque

• The Cartesian ego Descartes detected might not even exist

If the folk Cartesian story about justification is wrong, what’s left?

An alternative picture of justification is reliabilism: A belief is justified when it’s formed and maintained through reliable processes that tend to produce true beliefs. Justification isn’t about having some special subjective feeling of certainty or being able to introspectively verify your mental states. It’s about the process that produced the belief being a good one.

Consider how you actually form beliefs:

Visual belief: You look out the window and form the belief “It’s raining.”

• Your eyes receive light reflected off raindrops

• Your visual cortex processes this information

• Pattern recognition systems identify the characteristic appearance of rain

• Memory systems match this against stored representations of rain

• A belief state is formed: “It’s raining”

Is any of this process fundamentally dependent on having qualia or a Cartesian theater? It doesn’t seem like it. The process is:

1. Receive sensory input

2. Process that input through various computational mechanisms

3. Compare against stored patterns and previous experiences

4. Form a representation of the world state

5. Store this representation for future use

This is an information-processing story all the way down. And crucially, this process is reliable: it tends to produce true beliefs about whether it’s raining. That’s what makes the belief justified.

On the reliabilist account, justified beliefs don’t need to be certain. They just need to be formed by processes that usually get things right.

This actually matches our intuitions better than the Cartesian story. Consider:

You look out the window and see what appears to be rain. But it could be:

• A sprinkler system you forgot about

• A movie set creating artificial rain

• An elaborate hallucination (you’re on drugs, or having a stroke)

• A very realistic dream

• A brain-in-a-vat scenario where you’re being fed false sensory data

You can’t be certain it’s raining with the kind of absolute certainty Descartes wanted. But that’s fine. Your belief is still justified because it was formed by a reliable process. Most of the time when your visual system produces the representation “rain,” there really is rain. The process is reliable even if it’s not infallible.

What counts as a “reliable process”? Here are some features:

1. Correlation with truth: The process should produce true beliefs more often than false ones in the environments where it operates.

2. Appropriate sensitivity to evidence: The process should update beliefs when new evidence comes in, and it should be more confident when evidence is strong and less confident when evidence is weak.

3. Coherence: The process should produce beliefs that fit together consistently. If you believe “All mammals are warm-blooded” and “Whales are mammals,” your processes should lead you to believe “Whales are warm-blooded.”

4. Resistance to known error modes: The process should avoid systematic biases and known sources of error, or at least be able to correct for them.

5. Appropriate use of background knowledge: The process should integrate new information with relevant prior knowledge rather than treating each belief in isolation.

Could a machine implement processes with these features?

Let’s think about how a machine might form the belief “It’s raining”:

• Sensor input: A camera receives visual data, or a moisture sensor detects water.

• Pattern recognition: Neural networks or other algorithms process this data to identify rain patterns.

• Comparison with stored data: The system compares current inputs against training data or previous observations.

• Belief formation: The system forms an internal representation: “It’s raining.”

• Confidence assessment: Based on signal strength and pattern matching confidence, the system assigns a probability or confidence level.

Is this process reliable? Well, that depends on the specifics:

• How good are the sensors?

• How well-trained is the pattern recognition system?

• Does the system have enough relevant training data?

• Can it distinguish rain from sprinklers or other confounds?

• Does it update appropriately when conditions change?

These are all empirical questions. There’s no in-principle reason why a well-designed system couldn’t form beliefs through highly reliable processes. In fact, for some tasks, machine processes might be more reliable than human ones.

Under reliabilism, it looks like machines can in fact have justified, true beliefs. Therefore, it seems like machines can have knowledge.

The implication that humans might be incredibly complex biological machines rather than transcendent experiencing angels should hopefully be a little more understandable than when we started.

Conclusion

So I claim folk Cartesianism is not “obviously correct,” and a better story of the mind looks something like this:

• Humans do not have Cartesian egos, magical eyes floating outside of physical reality observing what happens in our minds like a movie.

• Human experience is deeply fallible as a source of information about our own minds, and even of the experience itself. We do not actually have a clear understanding of what’s happening in our own minds, or even a clear understanding of what we are currently experiencing at any given moment. “What it is like to be us” is not something our brains have special, ultimate access to. We have very strong evidence, but our minds are not displayed before us clearly like a movie.

• “True introspection” where we step back and look at our thought processes is not possible. Introspection is just another mental function that takes in inputs and gives outputs. We cannot actually see the wiring of our own minds. “Introspection” is actually just another informed guess about the world our brains make, this time based on the patterns we’ve observed in our own behavior. Introspection is like observing a natural phenomenon and making predictions about what will happen next. It’s not like looking in a mirror and clearly seeing ourselves as we really are.

• Human knowledge is based on publicly observable evidence and testable reasoning, not privileged access to private mental contents. Our beliefs are justified through the same kinds of information processing that machines can perform, drawing inferences from data, testing predictions, and refining models based on outcomes.

• The meaning of language is rooted in public use and social context, not private mental associations. We learn words through observable patterns of usage, not by linking them to ineffable subjective experiences. This means machines can acquire genuine understanding of language without needing a Cartesian theater.

• There is no fundamental barrier preventing machines from having beliefs, knowledge, or understanding. If our minds are physical information-processing systems (as physicalism suggests) and what matters is function rather than material (as functionalism suggests), then appropriately designed machines could replicate any cognitive capacity humans have.

I’ll close with a famous line from Derek Parfit on the reductionist view of personal identity, where he describes the experience of letting go of his sense of having a Cartesian ego and internal movie theater.

On the Reductionist view each person’s existence just involves the exercise of a brain and body, the doing of certain deeds, the thinking of certain thoughts, the occurrence of certain experiences, and so on…

Is the truth depressing? Some may find it so. But I find it liberating, and consoling. When I believed that my existence was such a further fact, I seemed imprisoned in myself. My life seemed like a glass tunnel, through which I was moving faster every year, and at the end of which there was darkness. When I changed my view, the walls of my glass tunnel disappeared. I now live in the open air. There is still a difference between my life and the lives of other people. But the difference is less. Other people are closer. I am less concerned about the rest of my own life, and more concerned about the lives of others.

“Let me see if I understand your thesis. You think we shouldn’t anthropomorphize people?” -Sidney Morgenbesser to BF Skinner

The 2020s have made my philosophy degree feel more worthwhile. Not only has social media made a ton of new people interested in philosophy, but the real world has also made many of the main questions in philosophy more relevant. AI especially has brought many philosophy topics into the mainstream, but it’s also polarized a lot of the debates. Back in 2011, I could say things like “The human mind is composed of natural processes, like everything else in the world, and therefore it could probably eventually be replicated by a machine, because in some sense it itself is a machine” and people would have long drawn-out conversations with me about whether that’s true or what that implies about language or experience or society. Now, when I say the same thing, it’s much more likely that I’ll get accused of “just buying into the recent AI hype” or that I’ve been tricked by secularism or capitalism or worse to deny the fundamental transcendent character of human minds. This issue used to be a pretty marginal question, but now it’s entered mainstream partisan debate. Sides have been taken.

What’s odd is that a lot of critics of the “minds are machines” view used to implicitly defend it. Back in the 2010s, there was a general worry about anthropocentrism. The belief that humans were in some way unique or separate from the natural world was regularly sneered at as “naive humanism” at best and a method of justifying exploitation of the natural world at worst. There was also a lot of talk about Cartesianism (the philosophy of René Descartes) as the origin of many of society’s problems. Specifically, mind-body dualism was accused of denigrating embodiment and elevating technocratic authoritarian reason. I always thought both criticisms went too far, but analytic philosophy had turned me into an anti-Cartesian naturalist, so I was able to nod along and contribute to these conversations where I could. Since then, attitudes toward Cartesianism have flipped. Many of the same people are now saying that AI cannot, by definition, ever have knowledge, because it does not have subjective first-person experience, or that it’s lacking some transcendent non-physical characteristic the human mind has; both implicitly Cartesian takes. Accusations of anthropocentrism have also been replaced with accusations of anthropomorphizing. A common line is that “everyone who actually knows how this technology works knows that it’s just linear algebra.1 The only people who believe it’s thinking are dummies who anthropomorphize it because it seems human.” I long for the 2010s when it was more common to hear people talk about a lot of different systems exhibiting some basic qualities of mind. I think back then people would be less likely to assume linear algebra couldn’t mimic some basic mental processes.

There’s a cluster of beliefs I associate with “folk Cartesianism” that even critics of Descartes often fall into. It’s incredibly common, to the point that people talk as if its implications are obvious without ever having directly thought about them before. I think this cluster of beliefs falls apart when you poke at it, and the fact that it’s implicit in so many debates about AI shows that it’s not being poked at enough. This post will be me poking at folk Cartesianism, and arguing for “anti-Cartesian naturalism” which is a widely-shared view among analytic philosophers of mind. Nothing I write here will be new or surprising to people knowledgeable about analytic philosophy. But I’ve been feeling the gulf between academic philosophy and mainstream discourse and want to contribute to building a bridge. This will be a long collection of the most popular and effective arguments for why folk Cartesianism is wrong, and how I think it’s harming the AI debate.

I have three goals here:

• You can identify what folk Cartesianism is, and when people are thinking like folk Cartesians.

• You come away no longer viewing folk Cartesianism as the obvious, common-sense view of the mind. You have a better understanding of the anti-Cartesian naturalist view, and philosophy of mind in general.

• You are more open to the idea that machines can replicate what happens in human minds. That minds are not magical, inexplicable substances separate from nature.

I’ve written this partly as an instruction manual to my past self I wish I had when I was younger, as my personal crash course in philosophy of mind 101 as it relates to AI.

Some of these points will be taken from my other posts on the same ideas. The “Naturalism” section will be repetitive for people who have read those already.

Contents

Part 1: Defining the problem

• Definitions

  • Folk Cartesianism

    I define folk Cartesianism as consisting of three common intuitions:

    1. There is a unified self that floats above mental processes and observes them

    2. We know our own internal experience with complete certainty

    3. The basis of all knowledge lies in our first-person subjective experience

    • The Cartesian view of the mind

  • Naturalism

    • Definition

    • Physicalism

    • Functionalism

    • Evolution

Part 2: Problems for Cartesianism

• “There is a unified self that floats above mental processes and observes them”

• “We know our own internal experience with complete certainty”

• “The basis of all knowledge lies in our first-person subjective experience”

Definitions

Folk Cartesianism

René Descartes inaugurated modern philosophy by attempting to establish a way to reach ultimate truth completely dependent on reason alone (rather than religious revelation). To do this, he looked for a solid, undeniable foundational fact he could start at. Everything that followed would be equally certain. He famously attempted to doubt all his beliefs, and found that the one belief he couldn’t doubt is that he himself existed. “I think, therefore I am.” From here he reconstructs all his other beliefs by building an intellectual ladder, with his belief in himself as the foundation. The next step is a pretty weird proof of God, followed by concluding that God wouldn’t deceive us about the external world.

Descartes’ explicitly and implicitly put forward many more specific claims about the mind, and his overall account matches how many everyday people imagine their minds working. “Folk Cartesians” mostly haven’t read Descartes, but have been influenced by how he and the broader culture talk about mind. So when I talk about “Cartesianism” here, I mean a cluster of concepts that are popular in everyday folk theories of the mind, not necessarily what literal Cartesian philosophers believe.

There are three key features of Cartesianism that I want to deny, and that are relevant for current conversations about AI.

1. There is a unified self that floats above mental processes and observes them

A “Cartesian ego” is a featureless observer hiding in the mind. This is the “I” in Descartes’ “I think therefore I am.” Cartesians imagine the mind as something like a movie on a screen, with the ego the silent observer. Critics call this the “Cartesian theater” model of the mind.

2. We know our own internal experience with complete certainty

Cartesians would say that we ourselves are egos. We observe our first-person experiences and thoughts. We have direct, clear access to them, in the way someone at a movie theater has access to the picture on the screen. The Cartesian ego also has direct, unmediated access to thought processes. We can turn our inner eye toward our thoughts in the way we would a piece of paper that lays out a step-by-step argument. We can observe the mental moves we’ve made and introspect on these thought patterns by just observing them directly. When we introspect, what’s happening is that we’re stepping back and observing the overall shape of our thought processes as they happen, like watching a line of dominos fall while standing above them.

3. The basis of all knowledge lies in our first-person subjective experience

Anything we could call real knowledge of the world has to come from the Cartesian ego observing and experiencing the world, or introspecting on its thought processes. A machine cannot have knowledge, for the same reason a line of dominos cannot have knowledge: there’s no Cartesian ego behind the scenes experiencing and judging the world. Thus, true artificial intelligence is impossible by definition, and many go farther and say AI is useless at what it’s trying to do. It can only ever imitate and copy knowledge. It cannot have knowledge of its own. It will always and only be glorified auto-complete. Even a machine that can handle and process large quantities of information cannot have knowledge, because the mind does more than information processing.

Many people apply this to our understanding of language as well. We only access the meaning of words via our first person subjective experience. I only know what “blue” or “tree” means because I have had subjective, first-person experiences of blue things and trees, and have learned to associate the words with those experiences. More complex concepts like “the national debt” are made up of simple concepts I’ve experienced like “money.” Any entity without a Cartesian ego cannot actually understand the meaning of words, because it cannot have these foundational experiences where it learns what words really mean.

Together, these three beliefs guide how many people think about their own minds, and how they think about AI. I will argue that they are incorrect.

The Cartesian view of the mind

The three of these paint an overall picture of the mind: a rich, internal theater where our egos live and experience things. Most of the important stuff that we consider “mind” happens here. Mathematical functions performed on our experiences and ideas happen a little behind the scenes, and we can perform those functions in our little theater as well (like when we hold numbers in our head while adding them together). Computers can replicate these mathematical functions, but they cannot replicate the internal theater. My goal for this post is to build some intuitions for the idea that this theater doesn’t exist. That we are not egos perceiving clear, unambiguous experiences, and that the mind is actually functions all the way down, where “function” is defined as any process that takes in inputs, performs computations on them, and gives an output.

Naturalism

Definition

I’m arguing for “anti-Cartesian naturalism.” Philosophical naturalism is a loose concept. It basically means “the world follows the laws of science, there is nothing magical happening.” Most analytic philosophers are naturalists, they just have strong disagreements about what counts as part of the natural world. There are three concepts associated with naturalism I adhere to in philosophy of mind:

• Physicalism

• Functionalism

• Evolution

I’ll summarize and give some basic arguments for each.

Physicalism

Physicalism is the view that the only things that have causal effect on physical objects are other things described or used by the laws of physics. Physicalism could be said to mean “Ultimately, what exists is the universal wave function.”

For something nonphysical to have causal effect on physical entities would violate our current best theories of physics. Anything new that has causal effects is likely to be an entity similar to what physics is already describing. Physics has been remarkably successful at describing the world, and physicists operate under the assumption that anything influence physical objects will itself also be governed by physical law.

These are some qualities that all physical entities seem to share, and that differentiates them from supernatural entities:

1. Spatiotemporal location: Physical things exist in space and time. They have locations, durations, and can be detected at specific coordinates. Supernatural entities are often conceived as existing “outside” space and time.

2. Causal structure and lawfulness: Physical things behave according to regular, mathematical laws that describe their interactions. These laws are:

   1. Discoverable through observation and experiment

   2. Consistent and predictable

   3. Expressible mathematically

   Supernatural things are often defined by their ability to violate or transcend these laws.

3. Detectability/measurability: Physical things can be measured, either directly or through their causal effects. They leave empirical traces. Dark matter counts as physical even though we don’t know what it is, because we detect its gravitational effects.

4. Public accessibility: Physical phenomena are intersubjectively observable. Multiple people can measure and verify them using instruments. They’re not private or accessible only to special individuals.

There are entities in the universe we don’t understand, like dark matter and energy, but we should still expect those entities to conform to these 4 patterns when we eventually learn more about them. While physicalists believe that many more things “exist” besides fundamental particles and fields, those things are ultimately just names for those fundamental particles and fields interacting in some way. Even something as abstract as “the national debt” can in some sense be reduced to the incredibly complex behavior of the physical particles that make up people on Earth.

Functionalism

From the Stanford Encyclopedia of Philosophy:

Functionalism in the philosophy of mind is the doctrine that what makes something a mental state of a particular type does not depend on its internal constitution, but rather on the way it functions, or the role it plays, in the system of which it is a part

Functionalism’s main claim is that mental states are defined by what they do, not what they’re made of. Among professional philosophers, functionalism is the most popular account of mental states.

A mathematical function takes an input and produces an output. Functionalists imagine mental states as being functions that take specific inputs and give outputs. A good example of something else like this is monetary value. “Twenty five cents’ worth of value” can exist in a quarter or a digital bank record of $0.25. What makes it the same monetary value is the function it plays in our broader system of exchange. When you transfer it to someone else, you can expect their behavior to be the same regardless of whether it’s delivered as a physical object or a digital exchange.

Similarly, the state of being in physical pain can happen in different animals with different physical brain structures, or even brains made of different physical substances, provided that their mental architecture reacts similarly (gives different outputs) to the same inputs. Just like the monetary value of twenty five cents can be instantiated in different physical systems (a metal coin or a computer), mental states as functions can also be instantiated in different physical objects.

If a system is able to reproduce all the same inputs and outputs as mental states as they occur in human minds, and is just made of different physical material, we should probably assume that it also has the same mental states as the human brain.

I don’t think current AI as it exists is conscious, but it is possible in principle for a machine made of silicon to be conscious like humans if it can reproduce the same inputs and outputs of our relevant mental states. It might be that at a certain level of complexity AI could also have mental states vastly alien to what humans can experience, but that we would still consider conscious experience.

An argument for functionalism, presuming physicalism

Suppose that we were able to replace exactly one neuron in my brain with a silicon chip that could send and receive exactly the same electrical impulses at exactly the same times as the neuron. Maybe there’s something about the substrate of silicon that prevents it from replacing all the functionality of a neuron, but let’s presume for now that it can send and receive all information the neuron receives. What would happen?

Importantly, if physicalism is true, nothing about my behavior would change. There wouldn’t be some immaterial soul or something similar swooping in to say “Hey, Andy’s brain has something strange in it. Something’s off!” The same electrical impulses would happen in my brain either way and cause me to behave in the same way. The same inputs would lead to the same outputs. If I looked at a tree and said, “I see a tree” before replacing a neuron with a chip, I would say the same afterward.

So if physicalism is true, my behavior would stay the same after my neuron is replaced with a chip. What would happen to my conscious experience of the world? Let’s imagine I’m staring at a tree (I’m using an example of my conscious experience of my field of vision because it’s easier to make an image). It seems like there are maybe 4 options, shown below:

• Option 1: No change - My conscious experience of the world doesn’t change.

• Option 2: A little missing - Maybe some small specific part of my field of vision or something similar goes blank.

• Option 3: Less consciousness - There’s less conscious experience in general. Maybe my experiences are more blurry or dim, like I’m in a dark room.

• Option 4: No consciousness - I no longer have conscious experiences.

We’ve established that my behavior wouldn’t change. Physicalism implies that the same inputs would lead to exactly the same outputs. This would make option 2, 3, and 4 very strange, because I wouldn’t react to my conscious experience changing, because my outward behavior would be exactly identical to the before case. If I looked out at a tree, and my conscious experience included a small piece of the picture missing, I would notice that and comment on it: “What’s going on? There’s a black dot in my field of vision.” If there were less conscious experience in general, I’d probably say “My vision is suddenly very blurry.” If there were no conscious experience, I’d say “Aaaah! I can’t see!” Because my behavior doesn’t change, I don’t say any of these things, I say “Look! A tree!” This would be very strange behavior if 2, 3, or 4 were true.

I trust my current self’s reports about his conscious experiences. My speech about my conscious experience matches the experiences I’m having. I should probably trust silicon neuron brain Andy’s reports as well.

Maybe it’s possible that my conscious experience would be disconnected somehow from my physical behavior, where I’d be helplessly losing consciousness as my physical body behaved normally as if it were a robot. Unconscious silicon brain Andy would take over as normal brain Andy helplessly lost consciousness and died. This would be very strange, because most of my day to day behavior is responding to my conscious experience. If you somehow removed my conscious experience’s ability to affect my physical behavior, I’d behave in very physically different ways, because I’d be missing a lot of important information. I wouldn’t react to heat or cold or anything in my field of vision. I might behave as if I were dead. It seems like it’s very likely that my conscious experience wouldn’t be affected by replacing 1 neuron with a silicon chip.

We could then go ahead and replace a second neuron with a silicon chip. For the same reason, my conscious experience wouldn’t change. We could build special chemical receptors and signals to replace other parts of my brain. As long as they had the same inputs and outputs on my behavior, I wouldn’t expect my conscious experience to change. At the end, I’d have the same mental states as before, but my brain would now be made mostly of silicon and other materials. This thought experiment is evidence that functionalism is correct: what makes something a mental state depends on its inputs and outputs, not on the specific physical material the system creating that mental state is made of.

So we’ve established that if physicalism is true, it seems likely whether my mind is in a certain conscious state is that the same inputs lead to the same outputs, not the internal constitution of what’s creating those inputs and outputs (a neuron vs. a silicon chip). Physicalism seems to imply functionalism.

Two other key ideas that together seem to imply functionalism:

• The Neuron Doctrine: The idea that the brain is made of discrete cells (neurons) that communicate through connections. This supports functionalism because it suggests the brain works through information processing between units, not through some special physical substance. This is a central tenet of modern neuroscience.

• The Church-Turing thesis: Any function that can be computed by any possible mechanical process can also be computed by a Turing machine. In other words, there’s a fundamental equivalence to all forms of computation. They’re all doing the “same thing” at a deep level.

If the mind is instantiated by a system of neurons taking inputs and sending outputs to each other (performing specific functions), and that this exact same type of thing can be instantiated in many different mediums, this seems to imply that everything we think of as mental can in principle happen in other systems that can perform computations.

It’s important to note that many non-physicalists are also functionalists, David Chalmers being the most famous example.

Evolution

If naturalism is true, the mind is a product of evolution. Like all other parts of all animals, the mind was subject to evolutionary pressures and natural selection. We should not expect that the mind came pre-made to access abstract truths about the world. Instead, for most of life’s history, minds mainly operated as machines to predict very local environments and specific situations, help the animal survive and reproduce, and (in more advanced brains) achieve and compete for social status (social status here being very broad, and including things like friendship and belonging). Given our evolutionary history, the real question is “Why do people believe true things?”

Dan Dennett had a great line about how “conscious human minds are more-or-less serial virtual machines implemented inefficiently on the parallel hardware that evolution has provided for us.” Put in more everyday language, our ability to do abstract reasoning, to follow a logical argument from premise to conclusion, to work through a math problem step-by-step, or to carefully consider the pros and cons of a decision, runs on top of brain hardware that wasn’t really designed for this kind of thinking.

Our brains evolved to handle dozens of things simultaneously: monitoring our surroundings for threats, maintaining balance, regulating body temperature, processing faces and voices, coordinating movement. This massively parallel processing is what kept our ancestors alive. In comparison, conscious, deliberate reasoning was rarely useful in the ancestral environment.

This helps explain why certain mental tasks feel so effortful. When you’re trying to hold multiple abstract ideas in your head at once, or forcing yourself to focus on something boring, you’re essentially running software that’s fighting against the grain of the hardware. Recognizing a friend’s face in a crowd or catching a ball requires vastly more computation, but feels effortless, because that’s what the parallel hardware excels at.

We shouldn’t expect our conscious minds to be perfectly rational machines, or to have some kind of magical access to higher realms of truth than what is available to other complex physical systems. We can obviously access truths that other systems can’t, but our access to these truths demands physical and evolutionary explanation.

This is my favorite Sora video, and AI video in general:

Watch that again, focusing on how the way the horse’s hooves move is so coherent and makes sense given what it’s doing.

This is obviously slop. It’s a meaningless short video that won’t benefit me at all beyond activating my dopamine for a moment. But it’s also a sign of pretty remarkable ability.

Sora wasn’t told the laws of physics, or how horses move, or what skateboards are. It was able to infer all this from scratch from intaking gigantic amounts of video data. By getting more and more data, and having more and more internal “knobs” it could adjust to respond to that data, it was eventually able to infer enough useful heuristics that it can create a convincing video of a horse riding skateboards. This is an especially great new video explanation of how this works:

There are a few things happening in AI right now that make objective assessments of its capability difficult. Among other things:

• People are constantly overhyping what it can currently do, leading to general skepticism when it doesn’t match up.

• The data center buildout has been massive, and headlines about the cost and impacts have led many to believe that AI needs to be a world-historic technology right this second to justify it.

It’s very easy to react to any new improvement in AI by just immediately identifying all the ways it doesn’t measure up to the buildout or the hype. There are also a lot of in-the-moment status games to win. But try to think back to yourself 5 years ago. Would you have been able to predict that AI video would be able to do this in 2025? Here was the best state of the art research on AI video at the time:

Now we can make videos like this:

Sora and chatbots both depend on the transformer architecture, the pattern-recognition and heuristics-generating tool that’s been the backbone of most AI progress over the last decade. Transformers take in huge amounts of training data, and adjust huge amounts of internal parameters to pick up on more and more subtle heuristics and patterns. This ability to detect such complex and nuanced patterns is on display in the horse video. There is no way you could personally code clearly-written rules into a machine to make it so generally knowledgeable about the subtleties of video and motion that you could just type “Make a horse jump a ramp” or “Make Spongebob get a ticket” and expect it to give a useful output. This is why deep learning is so useful. It’s a way to get a machine to build its own soup of internal heuristics for how to handle complex situations. This soup of heuristics both succeeds and fails in surprising ways. We don’t design the heuristics themselves, and don’t even know what they are or how they work, in the same way we don’t know how a lot of the heuristics in our own brains work. We can now make machines that can pick up on such subtle heuristics that they can infer how a horse would look skateboarding over a ramp to the point that their output looks photorealistic.

It seems obvious that if deep learning can pick up on the patterns required to make the horse video, it can probably be used in a lot of other more important places in society. If it can independently figure out some rough approximation of the laws of physics to the point that you can be tricked into thinking videos of complex motion are real, surely it can help with weather forecasting, or have a huge number of specific applications for simple problems in climate, or might soon pose some new emergent risks? Sora 2 makes it especially hard for me to understand how anyone can say that deep learning models are “just glorified auto-complete.”1 They seem to be able to develop some pretty useful internal heuristics that correlate pretty strongly with what’s actually up in the world. If “glorified autocomplete” systems can produce photorealistic videos of anyone in any describable situation, the term stops being useful as a way of predicting what models can and can’t do.

ChatGPT has only been with us for 3 years. The time from the release of Windows 95 to Windows 98. The jump in AI capabilities has seemed much more significant to me over these 3 years than the jump in usefulness from Windows 95 to 98. An AI having the ability to beat out TikTok and Instagram as the most-downloaded short form video app (even for a moment) would’ve been unthinkable… a few weeks ago? Now I’m seeing Sora videos flood Instagram Reels. AI beating the slop machines at their own game, even temporarily, just seems like another win for people in the “AI is shockingly capable, and the capabilities are improving fast, so this really really doesn’t seem like a fad” camp. I’m worried that a lot of people are getting distracted by the stupidity of the outputs to notice how strange of a world we’re living in now.

First, 2 definitions:

• Heuristics vs rules: Rules are absolute, fixed principles that must be followed to guarantee a specific outcome, while heuristics are "rules of thumb" flexible, practical shortcuts or educated guesses used for efficient problem-solving and decision-making, though they are not guaranteed to be optimal, perfect, or even correct.

• Deep learning: The specific type of AI that’s become dominant over the past decade, where instead of humans writing out explicit rules, we build large neural networks that learn their own internal rules or heuristics from data. These models stack many layers of simple mathematical functions, allowing them to automatically extract patterns from raw inputs (like pixels, audio, or text) and make increasingly complex decisions. Basically all debates about “AI” right now are debates about deep learning specifically.

Deep learning is a way to get a machine to build its own soup of internal heuristics for how to handle complex situations. This soup of heuristics both succeeds and fails in surprising ways. We don’t design the heuristics themselves, and don’t even know what they are or how they work, in the same way we don’t know how a lot of the heuristics in our own brains work. There’s a whole field of AI interpretability attempting to understand AI models’ internal heuristics. It’s hard!

Could you write a clear step-by-step algorithm to help a machine determine when a song feels nostalgic and when it doesn’t? Can you identify when a song feels nostalgic when you hear it, even if you don’t have the emotional reaction of feeling the nostalgia itself in the moment? If you can identify nostalgic songs, there are heuristics in your brain that you can’t articulate, and that maybe aren’t really articulable using our current language or methods of consciously thinking about the world.

Having some way to get a machine to build up similar inarticulable intuitions that lead to good results like this seems useful. The main benefit of deep learning is that before it, basically everything we used machines for had to be articulable in clear language and logic. If there were situations that we could build up tacit knowledge for how to handle, but couldn’t possibly write that knowledge down as clear instructions, machines couldn’t join us in navigating them. Now (with many bumps and failure points) they can.

A funny example of this is chicken sexing. There are no clear written rules for identifying whether a newly hatched chick is male or female. The average person looking at one would just take a wild guess. Something odd happens where, if humans train for months, they can achieve 97% accuracy in looking at a newly hatched chick and identifying whether it’s a male or female, but no one who can do this can articulate how they’re doing it, so they can’t just write an instruction manual new people can use to skip those months of training. This is a classic example of an internal soup of heuristics. Our brains develop some complex internal rules for what male vs. female newly hatched chick look like, but these rules aren’t actually available to our conscious mind. Until deep learning, we couldn’t build machines to do this task for us, because we couldn’t articulate what we’re doing when writing the machine’s code or working on its physical design.

If you train a deep learning model on enough images of chickens, it can also pick up on whatever opaque heuristics the humans are using. It also can’t articulate what those heuristics are, but whatever internal rules it developed are good enough that the machine can distinguish male from female chickens at 98.5% accuracy, beating out humans.

To get a sense of why and how deep learning models can develop these inarticulable heuristics, I really cannot emphasize enough that you should watch the 3Blue1Brown series on neural nets. It’s the best popularization of how deep learning works anywhere.

If a tool can develop heuristics that are inarticulable to itself and us, this implies 2 things:

• The number of possible use cases for it is gigantic. Computers took off so fast because they were so uniquely useful at solving problems that had clear logical steps we could articulate to them. This type of problem is very common. Deep learning models can help us with the other type of problem humans work on: places where we can’t articulate perfect rules to follow, but we know solutions when we see them, and experienced humans can develop tacit knowledge about a problem and solve it using intuition. There are a lot of examples of this type of problem as well.

• There will be a lot of weird ways these machines will fail, and they can’t be completely relied on in the way we could rely on computer programs with clear step-by-step instructions we could refine and test. This is the first time we’ve been consistently using machines with complex webs of rules and heuristics that we ourselves didn’t program into them. Instead of saying ChatGPT was programmed, it probably makes more sense to say something like it was “grown” in the lab during training. The potential for failures here is high (hence why AI models hallucinate and lie more consistently than normal computer programs), but the fact that they can develop their own heuristics at all makes them valuable even with a higher chance of messing up.

This is why many uses of deep learning (including ChatGPT) seem valuable to me, even though they can make a lot of mistakes.

I’m posting this for two reasons:

• I think a lot of people still don’t know how good AI art’s gotten. I’m surprised that I haven’t seen more people playing around with it, or making the type of images I personally like. If we have the same aesthetic taste, you’ve also probably thought a lot of popular AI art looks kind of ugly. If we enjoy the same vibes, I’m hoping this post will convince you that AI tools can help you make a lot of images you find interesting and fun.

• I get a lot of good feedback when I share AI images I’ve made on Twitter, so I want a single place to collect the ones I like the most. I’ll use this post as a place to dump basically every AI image I’ve made that I really like.

Obviously it’s a philosophical question whether I “made” these, or whether they count as art. If you prefer, we can say I “found” these images in MidJourney’s vector space. I’m excited that I found them in the same way I’d be excited about finding a neat rock. I don’t think much of my own creativity went into this, and I know it’s different from creating art myself. Curating AI art feels kind of like curating a Pinterest board. I think that can be a neat way to spend time to develop and show off your own taste even if you’re not putting any creative effort into the images themselves.

When I have time, I’ll go back through these and add the prompts I used for each as alt-text. If you’d like the prompts for any of these in the meantime please let me know!

There are 152 images here. According to the MIT Tech Review, this means that altogether I used about 185 Wh of electricity to make these. That’s about the same energy you use running a normal kitchen microwave for 12 minutes, or playing a gaming console like a PS5 for about an hour. The average American household uses this much electricity every 8 minutes.

Solarpunk landscape architect sketches

Some variation of “Solarpunk landscape architect sketch, watercolor” plus a lot of specific descriptions works well as a prompt here.

MidJourney now lets you animate every image:

Bit art of utopian architecture and communities

General weird architecture

Worlds made of weird textures

Album covers

Spheres

Brutalist monuments in forests

Animals (mostly pigs)

Cities and towns

Cafes & diners

Photorealistic/can you believe this is AI?

Art Nouveau

Old photographs

Maps & aerials

Surreal

Misc

Me: I think a lot of people are getting a lot of misinformation from TikTok, and I think it’d be better if TikTok didn’t exist, but I’d consider anyone who said that TikTok is completely useless to its users to be pretty goofy. I feel the same about chatbots.

User: How can you both think TikTok shouldn't exist and think that it's useful to its users, without using a pretty unique definition of useful?

I was surprised by this reply. It showed up in a lot of other places too. To me it seems obvious that something can be useful in specific situations, but still be bad overall. Here are some examples:

• Nuclear weapons: I would prefer a world where nuclear weapons didn’t exist, but nuclear weapons are extremely useful to the specific countries that possess them. It seems ridiculous to call nuclear weapons “useless.”

• Guns: I’d support strict gun control if I thought it worked, and I think guns often harm their owners, but there are a lot of specific cases where guns provide a lot of value to their owners. Guns aren’t “useless.”

• Alcohol: I think it’d be good for everyone to boycott alcohol, because the effects its having on the people addicted to it are so bad, but alcohol obviously also adds a lot of value to specific people’s lives. It’s not “useless.”

• TikTok: TikTok seems pretty addictive, distracting, and contains a lot of misinformation and weird ideological rabbit holes. It’s also fun to use and can teach you a lot of specific social cues and context. I think it’s overall bad for society, but it’s definitely not “useless” to the people on it.

When I say that it’s not debatable anymore that chatbots are sometimes useful, a lot of people read me as saying that it’s not debatable that chatbots are good overall, and that all critics of chatbots are crazy. I don’t believe this at all.

I think it’s totally rational to think that chatbots are bad overall. I could totally buy this criticism:

• Chatbots hallucinate a lot and give lots of incorrect information.

• The everyday person thinks they can rely on chatbots for correct information. They’re receiving more incorrect information than they would using traditional sources. This nets out to being bad overall.

• In addition, chatbots are giving people too many opportunities to cut corners on tasks they need to think about themselves. They’re very bad overall.

Any critic saying this could also agree that there are a lot of specific places chatbots are useful to people. For example, the only times I ask chatbots for contentious important information, I also ask them to provide sources, and I always check the sources. Most of the time, I’m asking them for much simpler lower stakes things, like sorting lists or doing simple back of the envelope calculations. Here’s a specific example where I get a lot of value out of using a chatbot. Any critic should be able to agree that those are examples of a chatbot being useful.

A lot of people do still talk as if chatbots cannot ever be useful for anything, period. People will question why I’d ever poke around on ChatGPT at all, or consider using it for literally anything. I think those people are obviously just wrong and silly. They’re making a fundamentally different claim than the people saying that chatbots are bad overall.

It’s no longer debatable that chatbots are useful in specific instances. It’s obviously very debatable whether chatbots and AI more broadly are good for the world overall. Clarifying that these are two different claims seems important if we want conversations to go well.

This is wrong. Most people aren’t super knowledgeable about a lot of specific facts about the world (only half of Americans can name the 3 branches of government) but they’re intelligent when it comes to their own lives and the areas they work and spend time in. We should expect the average person to struggle with factual questions about abstract ideas and far-off events, but not so much about what’s right in front of them day to day.

If a claim about how society works implies that most people are incredibly stupid, much more stupid than anyone I encounter in my day to day life, I dismiss it. This simple test kills a lot of big claims about how the world works. I’ve been applying it in a lot of AI conversations recently. I’ve written about this a bit before but want to go into more detail.

Here’s a common claim that I think fails my test: “The reason Americans are so unhealthy is that doctors don’t tell people about healthy diets.”

I think most people know what’s considered healthy food. They maybe wouldn’t be able to perfectly break down ideal ratios of macronutrients, but they have a rough idea. The average person whose bad diet is making them unhealthy would probably be able to point to the bad diet as part of the problem. If I walked up to the average person and asked them to make an ideal meal plan for themselves to be maximally healthy, I think most people would do a decent job.

Stefan Schubert makes a similar observation about what he calls sleepwalk bias:

When we predict the future, we often seem to underestimate the degree to which people will act to avoid adverse outcomes. Examples include Marx's prediction that the ruling classes would fail to act to avert a bloody revolution, predictions of environmental disasters and resource constraints, y2K, etc. In most or all of these cases, there could have been a catastrophe, if people had not acted with determination and ingenuity to prevent it. But when pressed, people often do that, and it seems that we often fail to take that into account when making predictions. In other words: too often we postulate that people will sleepwalk into a disaster. Call this sleepwalk bias.

I often use the idea of sleepwalk bias in conversations. However, what I’m pointing at here is a much more extreme example of assuming everyone is stupid about even normal everyday experiences, so I think it needs its own name. I’m calling it my "Are you presuming most people are stupid?" test.

Subscribe now

AI

There are a few claims about AI floating around that fail my test.

I was motivated to write this in response to this Time article: ChatGPT May Be Eroding Critical Thinking Skills, According to a New MIT Study. There are a few points in this article that break my rule.

Kosmyna, who has been a full-time research scientist at the MIT Media Lab since 2021, wanted to specifically explore the impacts of using AI for schoolwork, because more and more students are using AI. So she and her colleagues instructed subjects to write 20-minute essays based on SAT prompts, including about the ethics of philanthropy1 and the pitfalls of having too many choices.

The group that wrote essays using ChatGPT all delivered extremely similar essays that lacked original thought, relying on the same expressions and ideas. Two English teachers who assessed the essays called them largely “soulless.” The EEGs revealed low executive control and attentional engagement. And by their third essay, many of the writers simply gave the prompt to ChatGPT and had it do almost all of the work. “It was more like, ‘just give me the essay, refine this sentence, edit it, and I’m done,’” Kosmyna says.

What is this article actually telling us that the average person doesn’t already know? These seem to be the claims:

• If you use a talking robot to write your essay for you, you won’t learn as much about the topic compared to writing the essay yourself.

• Having a talking robot easily available to you makes you more likely to cheat on essay assignments.

Students using ChatGPT to write their essays for them aren’t stupid about what’s happening. Similar to students who just Google to find answers to homework problems, they’re aware that they’re making a trade-off between actual learning and saving time. This article is presuming that students are somehow blind to the idea that copying work from other places means they don’t actually learn. The average student isn’t like that. They make bad decisions when they cheat using talking robots, but they know what they’re doing.

Here’s another quote from the article:

The MIT Media Lab has recently devoted significant resources to studying different impacts of generative AI tools. Studies from earlier this year, for example, found that generally, the more time users spend talking to ChatGPT, the lonelier they feel.

It’s hard for me to imagine walking up to someone I don’t know and saying “Hey, spending a lot of time staring at your screen talking to a robot instead of interacting with real people can make you feel lonely. The experience itself can be somewhat alienating because the robot doesn’t feel human.” I don’t know how you could assume this is useful unless you assume the average person is really stupid. Would you feel comfortable telling a stranger this? Would you be able to say it in a way that isn’t demeaning?

To be clear, I think it’s good to conduct a lot of studies on “obvious” questions. My issue is with the reporting, not the original studies.

Another big claim that fails my test is that AI chatbots are useless. 10% of the world are now choosing to use them weekly. If they were useless, this would mean that 10% of the world is so stupid that they can’t tell that this tool they’re using every single week isn’t providing any value to them at all. There’s basically nothing else like this that people interact with regularly. You might think that social media like TikTok is bad for people, but it’s not “useless.” Users have fun or learn interesting facts or subtle social vibes from the TikTok videos they watch. You can criticize AI and think it’s net bad to use, but that’s a different claim from saying it’s useless. When I hear people say that AI chatbots are useless, it’s hard not to read it as a claim that almost everyone is incredibly stupid.

There’s too much of this way of talking in AI conversations. There are a lot of great criticisms of AI and chatbots, and real reasons to worry. I think that students cheating with ChatGPT is a gigantic crisis in education without clear solutions. But when people talk as if everyone using chatbots is incredibly stupid, and that people exposed to this technology are blind to the simple obvious trade-offs involved in specific situations, I come away with less respect for them. It seems like they underestimate the average person in a way that shows a lack of curiosity, or a tendency to steamroll other people’s experiences if they’re having a slightly different reaction to new technology.

Some popular written AI x-risk arguments are way too overconfident. The authors seem to be coasting off the social status their confidence about a new tech is giving them. Writing like this gives the impression that all arguments for AI risk are based on pie in the sky wild inferences that give the speaker social sanction to override normal rules in debate, and maybe life more broadly. I think the core AI risk case is pretty simple and convincing, to the point that I don’t really understand how people can just write it off, but I also think the case is often represented badly and it’s understandable that a lot of people won’t consider it because of the sometimes goofy behavior of some of its loudest representatives. This list will be me trying to separate the goofier takes from better arguments for what I think is a real and serious risk.

I put the odds of catastrophic risk from AI in my lifetime as low, but not low enough not to worry and think about. Anything above a 1% chance of human extinction seems high enough that most people should engage with the basic arguments.

Even if you don’t buy the risk case, writing on AI risk often brings in enough interesting ideas about technology, history, nature, evolution, and the future to be worth reading.

I’m very aware this is all very speculative, which is why I’ve made this a separate post. I don’t want to imply that anything I’ve written about chatbots and the environment or how AI works should give me any credibility on risks from advanced AI. This is basically a completely separate field of inquiry that I’m truly just some guy in. My only authority here is that I think I have good taste in reading material, and I’ve been reading talking and thinking about AI risk for over a decade.

Right now this list is pretty short. I’ll be adding to it as I pull together other materials. I’m trying to be pretty picky and selective. I’ve listed the resources in the order I’d consume them.

Articles

• On those undefeatable arguments for AI doom

  • I’m starting with this one to throw cold water on specific ways of thinking about AI catastrophe using fuzzy inferences that lead to drastic overconfidence.

• The 80,000 Hours profile on preventing an AI-related catastrophe

  • In my opinion the single best introduction to AI catastrophic risk.

• Matt Reardon’s The bone-simple case for AI x-risk

• Michael Nielson’s How to be a wise optimist about science and technology? and Notes on existential risk from artificial superintelligence

• Joe Carlsmith’s Existential risks from power-seeking AI

• Jacob Steinhardt’s series More is different for AI

• Holden Karnofsky’s Cold Takes series, which goes back and forth between takes I think are straightforward and convincing and takes I think are off the wall and goofy goober.

• Katja Grace’s Counterarguments to the Basic AI X-risk case

Classic texts

• Not AI, but related: Von Neumann’s Can we survive technology?

1. Ground truths everyone needs to agree on. These are the foundational ideas someone needs to accept for their opinions to be taken seriously. There are incorrect ideas circling around AI debates that need to be rejected before serious conversations can happen.

2. Useful rules for debates to make sure they don’t get stuck or confused.

3. Ideas I’d like taken seriously in any discussions about AI. Many reasonable people disagree with each, but each has enough serious people and arguments behind it that it deserves a seat at the table and shouldn’t be written off.

4. A rant about how tribal AI conversations have become.

I wrote this partly as an imagined instruction manual to my past self in my first year of college. I’d want past me to know what the most useful and exciting ideas are for thinking well about AI. In college I enjoyed reading long meandering blog posts way above their authors’ actual level of expertise, so I’m writing one of my own. This is why I’ve added long sections that explain Quine or authors writing about technological determinism. I’d want past me to know a lot about both.

Everything here is a one-off take so you can bounce around and only read what’s interesting to you. Some parts of this will be copied from earlier posts, especially my post on philosophy and AI.

Subscribe now

Contents

• 1: Ground truths everyone needs to agree on

  • Chatbots aren’t “useless”

  • Other people are smart

  • Tools to “replace thinking” sometimes help us think more deeply

  • The line between glorified auto-complete (next-token prediction) and real intelligence is blurry

  • Searle’s Chinese Room argument doesn’t show that deep learning models can’t think

  • Individual chatbot prompts’ environmental impact is small and comparable to everything else we do online

• 2: Useful rules for debates

  • Always include an “AI is capable/incapable” axis in addition to a “Pro” and “Anti” axis

  • No hyperbolic adjectives

  • Clarify when the complaint is that AI is empowering people too much

  • Clarify when AI is just a continuation of an existing tool or practice

  • Before people make big claims about what chatbots can and can’t do, they need to try using one

• 3: Ideas I’d like taken seriously

  • We’re already in an insane technological future

  • Well-respected mainstream ideas in analytic philosophy

  • The basic AI risk case

  • Excitement about new technology

  • Wariness about new technology

• 4: A final rant: AI discourse is full of tribal memes

  • People speak in post hoc rationalizations

  • Ways this shows up in AI discourse

1: Ground truths everyone needs to agree on

These are the facts about AI that are so important and obvious that we should write off the opinions of the people who deny them. It’s understandable to be confused about them when you first learn about AI, but having lots of opportunities to learn and still confidently making these errors is a sign that you’re willfully ignorant and trying to throw off the conversation.

Chatbots aren’t “useless”

Everyone needs to acknowledge that chatbots are sometimes useful to people. I don’t mean they’re perfect or they never hallucinate or they’re always preferable to a human. I just mean they’re one tool among many that can in particular circumstances make your life better. YouTube and TikTok and Instagram and Minecraft are also “useful.” None of these are perfect, each has specific use cases and things they’re bad at. The same goes for chatbots.

This seems obvious. However, there are enough people with large audiences making sweeping hyperbolic claims that AI is completely useless that it’s worth clarifying.

A simple proof that chatbots aren’t useless

A few days ago I was reorganizing the physics videos on my website. I already had them sorted on my YouTube channel. Instead of typing the names of each video or copying them one by one into a new list, I just took a screenshot of my YouTube playlist, gave it to ChatGPT, and asked “Please make a bullet point list of each, capitalizing only the first letter of each title, and removing “ - IB physics” from the end of each. I have about 200 videos, so this saved me at minimum half an hour of meaningless work.

I shared this online and got a reply from another physics teacher:

If “useless” means “something that cannot produce useful results and save you time” then ChatGPT is proved conclusively to be not useless.

Ways AI has been useful to people

Here’s a list of posts about how people use AI:

• Mine

• Andrej Karpathy

• Sean Goedecke

• Simon Wilson

If you read these and still think AI is useless, you have wildly high expectations for it that we don’t hold for any other technology. AI isn’t as good as having a personal smart human helper do anything I want, but it’s at least as useful to me as YouTube or Wikipedia or my cell phone’s camera. Each other tool also has lots of flaws and limitations, but also obvious places where it helps me.

It’s easy to criticize AI and find ways it can be harmful without calling it useless. It’s similar to cars. You can believe that cars are bad for society and create a lot of complex problems without believing cars are completely useless. Their usefulness is often the reason why the problems show up!

Other people are smart

Critics of chatbots sometimes talk as if most people aren’t smart. They’ll talk about how everyone using ChatGPT has been brainwashed into not noticing that it’s useless and can’t possibly provide value. A billion people use ChatGPT weekly. This is basically saying all of those billion people are too stupid to notice they’re using an app that produces gibberish, and never adds value to their lives.

You can think AI is bad for society without saying it’s completely useless to all the billion people using it. At some point this just shows a failure to respect the pluralism of other people’s tastes and goals for themselves.

I think TikTok is probably bad for society, but I understand that for a lot of people it’s provided a lot of subtle nuanced benefits you can only understand by using it and living in their specific social situations. It can teach you social cadences and give you glimpses into different lives and entertain you and help you keep up with popular jokes and memes. TikTok is useful for a lot of people, even if on net I think it’s bad. I know this because I’m pluralistic enough to defer to the large groups of people using it on whether it’s useful to them, even if I don’t get much value out of it.

People who aren’t using AI also often have great reasons to not use it. Domain experts can recognize where chatbots are too prone to hallucinations to help. I always assume these people know what’s up in their fields and can identify AI’s limitations. This is a great recent piece by a physicist on his negative experiences using AI for science and how it’s currently being overhyped in specific places. Again, it’s pretty easy to just assume most people are smart and can work out for themselves where AI is and isn’t useful to them. But assuming that every last AI user isn’t noticing that it’s not adding any value to their lives is deeply disrespectful to the judgment and values of an eighth of the world.

Tools to “replace thinking” sometimes help us think more deeply

Some people say AI is a “substitute for actual thinking.” There are a lot of serious examples of this, like students using chatbots to cheat. But some people take this farther and say that it’s always bad to “replace your thinking” with AI, even for mundane things like coming up with recipes or checking your writing for grammar issues or coding something simple for a website.

This specific criticism is too strong, because it applies to and condemns most useful technology and social practices.

Whitehead has a great simple quote I think of often:

Civilization advances by extending the number of important operations which we can perform without thinking of them.

If I want food, I “replace my thinking” by Googling a recipe and making a shopping list based on the ingredients. Google Maps removes my need to think about how to navigate on a road trip. Any philosophy book removes my need to think hard enough to arrive at the idea from first principles. A movie removes my need to make up a story in my head. Each of these are in some ways “replacing my thinking” with a technological experience that does the thinking for me.

Each of these also frees me from monotony and lets me get to the higher level thinking I’d rather be doing.

Imagine that movies didn’t exist, but I still want the higher-level thoughts and experiences that come with them. I’d have to put in the work of imagining a movie in my head. To get to the higher level thoughts, I’d have to climb all these additional steps:

Now I have access to movies again. They “do my thinking for me” and allow me to jump to what I actually want: a more complex experience of characters, vibes, and general commentary and emotional intuitions about life.

What I find meaningful and good about life exists at these higher levels. AI tools are a way to replace simple monotonous thinking tasks that won’t make my life better, like doing simple back of the envelope calculations or finding recipes online or web design or summarizing difficult technical papers I don’t have time to read. This helps me get to where I actually want to be: in a place where I can do more during the day and leave more room for the experiences that make life worthwhile.

It’s good to use technology to “replace” this type of thinking. It’s one of the things technology’s for.

Some people replace more important experiences with AI. They form relationships with AI chatbots or use AI for serious therapy or summarize books that they’d get a lot more out of if they actual read. This is irresponsible and should receive some social pushback, but in my experience it’s not the way most people are using AI.

The line between glorified auto-complete (next-token prediction) and real intelligence is blurry

LLMs are trained to predict the next word (or more precisely, the next token, which may not be a full word) in a string of text. The feedback they receive during training compares their guesses to the actual next words.

A common criticism of LLMs is that they’re “just glorified auto-complete” because they’re trained to predict the next word. I think people who say this don’t understand that next-token prediction is just the part of the model that receives feedback about whether it’s getting the word right. It takes this feedback and sends it back through over 1 trillion parameters to adjust its intuitions, eventually detecting extremely subtle patterns to build what looks a lot like a model of the world. This and this video are great explanations of how this feedback happens and how an LLM might come to build a model of the world using that feedback.

Next-token prediction is similar to evolutionary natural selection. Both are simple algorithms with simple outputs. Next-token prediction says “This word is more likely to be correct, that one is less likely to be correct” and tells the trillions of parameters of the AI to adjust accordingly. Each parameter is an individual numerical value adjusted during AI training. Natural selection says “This gene survived and reproduced, that one didn’t” and tells the species’ DNA to adjust accordingly. However, these simple inputs are both acted on by an unbelievably complex reality. For next-token prediction, all language and written knowledge of the world. For natural selection, all the complexities of nature, specific environments, and group competition. We can think of both simple algorithms as doors that allow the complexity of the world to enter and be used by the internal system of the AI or animal. The door itself can be small and simple, what matters is that the room inside is large enough to hold enough information to model the world. The AI’s parameters and the animal’s genes use an extremely simple language to save and record the information: for the AI, specific number values for its parameters, and for genes, the four nucleotides of DNA. The reason these simple values are able to capture so much of the outside world’s complexity and subtlety is that there are so many of them. The AI has trillions of parameters, and the animal has billions of nucleotide pairs. Simply by having an unbelievably large number of individually simple stores of information, AI parameters and DNA nucleotides are able to create entities that can respond to the world’s complexity with a shocking amount of subtlety. Just as evolution has used the simple algorithm of natural selection to build us unbelievably sophisticated bodies and brains that can react to the world, AI can use the simple algorithm of next-token prediction to build extremely complex internal systems to understand the world.

To get an intuition about how AI is trained to predict the next token and how that can build an inner world of associations and understanding, I strongly recommend the full 3Blue1Brown video series on neural networks.

Ilya Sutskever gave a brilliant example of why an AI trained to predict the next word will be incentivized to develop a world model and methods of reasoning:

So, I’d like to take a small detour and to give an analogy that will hopefully clarify why more accurate prediction of the next word leads to more understanding, real understanding. Let’s consider an example. Say you read a detective novel. It’s like complicated plot, a storyline, different characters, lots of events, mysteries like clues, it’s unclear. Then, let’s say that at the last page of the book, the detective has gathered all the clues, gathered all the people and saying, “okay, I’m going to reveal the identity of whoever committed the crime and that person’s name is”. Predict that word. Predict that word, exactly. My goodness. Right? Yeah, right. Now, there are many different words. But predicting those words better and better and better, the understanding of the text keeps on increasing. GPT-4 predicts the next word better.

Predicting the detective’s next word here clearly requires some kind of ability to model what is happening in the book. That model could be different from how human minds work, but some kind of model would be necessary.

Some people think that AI only memorizes canned responses to possible questions. This is not true. We can test this by asking AI some radically new question that it could not possibly have been trained to recognize, and see what happens. If you don’t think that AI “truly understands” the language it’s working with, it should be easy to come up with questions like this that clearly demonstrate that it’s just parroting words. Here’s a prompt I used to see if it can pick up on a lot of context

• Prompt: Can you write me a broadway musical (with clearly labeled verses, choruses, bridges, etc.) number about Donald Rumsfeld preparing to invade Iraq while keeping it hidden that he's secretly Scooby-Doo? Please don't use any overly general language and add a lot of extremely specific references to both his life at the time, America and Iraq, and Scooby-Doo. Make the rhyme schemes complex.

• Read ChatGPT’s answer here

My big question for hardcore AI skeptics is “What question could a chatbot answer well that would convince you it isn’t just a stochastic parrot?” And then ask Gemini the specific question. Once AI can give a good answer to literally any question, it starts to seem pointless to debate whether it “really, truly understands” the words we’re asking it to generate. Even if it doesn’t, it’s perfectly mimicking a system that does, so we’re getting the same value anyway.

Searle’s Chinese Room argument doesn’t show that deep learning models can’t think

This is a very specific point, but it comes up enough that it’s worth including. A surprising number of people bring up the philosopher Searle’s Chinese Room argument as a reason why computers can’t “think” in the way people do, or at least that the Turing Test (where a computer can successfully trick a human into thinking it’s also a human through text conversation) is insufficient to show that a computer actually understands language. The argument is simple:

Imagine a native English speaker who knows no Chinese locked in a room full of boxes of Chinese symbols (a data base) together with a book of instructions for manipulating the symbols (the program). Imagine that people outside the room send in other Chinese symbols which, unknown to the person in the room, are questions in Chinese (the input). And imagine that by following the instructions in the program the man in the room is able to pass out Chinese symbols which are correct answers to the questions (the output). The program enables the person in the room to pass the Turing Test for understanding Chinese but he does not understand a word of Chinese.

Searle is trying to show that a computer that responds to simple inputs and outputs doesn’t have real understanding of language in the way that humans do. He’s making a specific point about syntax and semantics. Before going on, it’s important to clarify what those are.

Syntax and semantics

• Syntax: The structure of a language: how words and symbols are correctly arranged. English speakers have background understanding of syntax even if they don’t always know the formal rules. Every English speaker knows that adjectives come before the nouns they describe, so we can say “A green apple is on a table” and can’t say “An apple green is on the table.”

• Semantics: The meaning encoded in language, from individual words to entire discourses. It's how we understand what words refer to, what sentences assert, and how these combine to convey information and intent. For instance, every English speaker knows that "The child ate the cookie" describes an action performed by the child upon the cookie, a meaning distinct from "The cookie ate the child," which, while both use the same words and correct syntax, have radically different meanings

A great example of the difference between the two is Noam Chomsky’s famous example sentence: "Colorless green ideas sleep furiously." This sentence has correct syntax: it follows the grammatical rules of English. It’s understood by all English speakers, whether or not they can state the rule itself, that adjectives go before the noun they’re modifying, and the subject for the verb comes before the verb. The sentence follows the structure (Adjective-Adjective-Noun-Verb-Adverb), which is a correct way of ordering words in English.

If we change the word order to "Colorless green sleep ideas furiously," the string becomes ungrammatical. It no longer follows English syntactic rules for declarative sentences (the subject-verb arrangement is disrupted), and it sounds like a disorganized jumble of words rather than a sentence.

Despite its syntactical correctness, the sentence is semantically meaningless. The actual meanings of the words don’t latch onto anything in the world we’re familiar with.

Chomsky's original sentence perfectly illustrates his point that syntax can be independent of semantics. A sentence can be structurally correct even if it fails to convey a sensible or logical meaning.

Searle claims computers can understand syntax but not semantics

Searle wrote years after making the argument:

I demonstrated years ago with the so-called Chinese Room Argument that the implementation of the computer program is not by itself sufficient for consciousness or intentionality. Computation is defined purely formally or syntactically, whereas minds have actual mental or semantic contents, and we cannot get from syntactical to the semantic just by having the syntactical operations and nothing else. To put this point slightly more technically, the notion “same implemented program” defines an equivalence class that is specified independently of any specific physical realization. But such a specification necessarily leaves out the biologically specific powers of the brain to cause cognitive processes. A system, me, for example, would not acquire an understanding of Chinese just by going through the steps of a computer program that simulated the behavior of a Chinese speaker.

Searle’s argument is a useful clarification of where traditional computing is lacking compared to the human brain. Simply following rote syntactical rules cannot give you a real understanding of the semantics of language. There’s an interesting debate about how valid this is about traditional computer reasoning (see this old podcast interview with Dan Dennett for some criticisms) but it pretty conclusively does not apply to the way deep learning models acquire knowledge.

Deep learning models are semantic meaning-catching machines

The magic of deep learning (and specifically the transformer architecture underlying it) is that it’s basically perfectly designed to give an AI model many many many opportunities to detect and build up subtle inarticulable intuitions about, and connections between, words, sentences, and (depending on your definition) concepts. This happens because transformers don't just look at words in isolation; they can weigh the importance of every word in relation to all other words in a passage, no matter how far apart. Imagine it learning, over billions of examples, which words tend to signal importance to others, or how sentence structures subtly shift meaning. This ability, combined with processing information through many layers, each building more complex understanding upon the last, allows the AI to develop a rich, internal "feel" for language. It's not explicitly taught definitions, but it discerns these intricate patterns from the sheer volume of text it analyzes, leading to a powerful, almost intuitive grasp of how ideas connect and flow.

To do a deep dive on this and get a lot of useful visuals on how deep learning models actually capture the subtle hard-to-articulate rules and connections between concepts, this video (and all the previous ones) from 3Blue1Brown is the best that I know:

The Chinese Room argument updated for deep learning would look something like this:

A man stands in a room with almost all writing that has ever occurred in Chinese. He has completely perfect superhuman memory, and trillions of years to read. Every time he is exposed to a new Chinese character, he stores hundreds of thousands of subtle intuitions about all the other places it’s been used in all Chinese writing, and as these trillions of years pass, the sheer density and interconnectedness of these hundreds of thousands of intuitions for every conceivable context of every character begin to form an incredibly rich and nuanced internal model. This model isn't just about statistical co-occurrence anymore; it starts to map the entire conceptual landscape embedded within the totality of Chinese writing. His 'intuitions' become so deeply interwoven and contextually refined that they allow him to not only predict and generate text flawlessly but also to grasp the underlying meanings, the relationships between abstract concepts, and the intentions behind the communications. Effectively, by perfectly internalizing how the language maps to the universe of ideas and situations described in all that writing, he has, piece by piece, constructed genuine semantic understanding from the patterns of use.

It seems like deep learning models are able to capture a lot of what we mean by semantic understanding. Searle’s Chinese Room may have been a good response to specific types of AI in the past (“good old-fashioned AI”) but it fails as a metaphor for deep learning, and doesn’t show that deep learning models can’t genuinely understand the information they take in and produce.

Everything’s ultimately rote or random

Some people run with Searle’s example to say something more extreme: intelligence can’t exist in physical systems that just follow rote rules, no matter how complex these rules are. Machines can never achieve true intelligence, because they’re just following rules and processing inputs and outputs, like the man in the room following inputs and outputs but never truly understanding what the Chinese symbols say.

Ultimately everything in the universe is either rote (where a specific input leads to a specific output) or random. This is true by definition. If an output can be perfectly predicted based on an input, it’s rote. If not, it’s random. There’s a debate about whether anything in the universe is truly random. Bell’s Inequality says yes, so let’s assume there is. In addition to the Chinese rule books, we can imagine the man occasionally spinning a roulette wheel of characters and sending a random one out. Chinese rooms and roulette wheels make up the whole of existence.

Unless you think something fundamentally magic is happening in the human brain, our minds and understanding ultimately have to be based on some incredibly complicated system of rote rules and random results, Chinese rooms and roulette wheels. A machine is also made of one or both of these, so the argument that machines can never understand things in the way a human brain does, because all machines are Chinese rooms, fails, because the human brain is a series of Chinese rooms and roulette wheels as well.

Individual chatbot prompts’ environmental impact is small and comparable to everything else we do online

I’ve been making this point over and over again recently so I’ll spare you by just linking to the post. You shouldn’t worry that your individual use of chatbots or image generators is harming the planet.

2: Useful rules for debates

These are rules I’d like every AI debate to follow to get more clarity and avoid pointless misunderstandings.

Always include an “AI is capable/incapable” axis in addition to a “Pro” and “Anti” axis

People who are “pro” or “anti” AI for wildly different reasons are lumped together in AI debates. If we organize people by how good or bad they think AI is, it looks like this:

A lot of conversations about who is “pro” or “anti” AI ignore differences in how capable people think the models are. It’s useful to clarify at the beginning of any conversation or debate not just how good or bad each person thinks AI is, but also how capable or incapable they think it is or can be.

A lot of media interviews introduce people as “pro” or “anti” AI. It would be clunky, but I wish people would always explicitly clarify at the beginning where they are on the capability axis as well.

No hyperbolic adjectives

I don’t know why AI talk draws so much hyperbole. I don’t mean extreme views. I totally understand having extreme positive and negative views about AI. I mean something more specific.

There’s a way of talking I call “hyperbolic adjectives” where a speaker wants to emphasize how good or bad something is, and just starts talking as if they’re reading from a thesaurus list of positive or negative adjectives. People say things like “AI is a planet-destroying plagiarism and gibberish machine that destroys the very concept of meaning and intentionality and is fundamentally fascist.” This feels antithetical to actual conversation. They could just say “I’m against AI because it’s really bad for the planet, doesn’t produce useful information, undermines a lot of our background trust in language, and enables way more authoritarian dynamics between capital and labor.” This makes all the same points but doesn’t explode in emotionally charged name-calling, and gives the other speaker more clear specific claims to respond to. Removing the emotional charge clarifies your own thinking as well. Do you actually think chatbots are some fundamental assault on meaningful language? Are you able to say that in a calm boring direct way and still have confidence that it’s true?

Tyler Cowen calls this way of talking pushing the button:

When I see people writing sentences of this kind, I imagine them pressing a little button which makes them temporarily less intelligent. Because, indeed, that is how one’s brain responds when one employs this kind of emotionally charged rhetoric.

As you go through life and read various writers, I want you to keep this idea of the button in mind. As you are reading, think "Ah, he [she] is pressing the button now!"

Clarify when the complaint is that AI is empowering people too much

I've heard people say "Chatbots are causing a massive student cheating epidemic. Of course, the AI companies didn't think of this before they unleashed them on us.” The complaint here is that the companies made their free product way way way too good and useful.

Genuine freedom is an important political value. We should always aim to maximize people’s freedom and access to tools they want to use, unless those tools directly give them new methods of taking others’ freedoms away. The value of freedom should be mostly independent of how people are using it. Freedom means the ability to go against other people’s standards of good and bad personal behavior.

Students have been given a very consistently correct general question answering tool. We should see this as good. It’s expanded their freedom and empowered them. Some of them are choosing to use their freedom in bad ways, by cheating. I see this as similar to choosing to use freedom of speech badly. The solution here and in most political problems is to create voluntary structures to manage the problems that come with freedom, not to restrict freedom in the first place.

People in AI debates sometimes go back and forth between claims that AI is too stupid to be useful, and AI is so powerful and effective at answering complex questions that it shouldn’t be given to students, because it’s too easy for them to use it to cheat. It’s weird to go back and forth between saying AI is too useless and too useful. “AI produces useless slop that also consistently gets A’s in college-level essays” doesn’t make sense. If it’s useless, we should try to limit it. If it’s useful, it’s by definition empowering its users and we should be excited about that, and come up with voluntary structures its users can participate in to manage the problems that come from being empowered to do whatever they want.

We should want to respect other people’s freedoms and pluralistic desires for themselves. Claims that AI is inherently bad because it’s empowering everyday people too much (to do sometimes goofy things) is a general condemnation of freedom and empowerment more broadly. We should reject this attitude in debates about AI.

Clarify when AI is just a continuation of an existing tool or practice

I was surprised at how negatively many people reacted to the ChatGPT Studio Ghibli images. I can understand finding them tacky, or boring, or repetitive (I did personally like them a lot), but some people took it way further to say that this was a unique attack on the dignity of Ghibli movies.

What I found strange about this was that the exact criticism could be made of poorly done fan art. There’s a lot of bad fan art of Studio Ghibli that also “disrespects” the source material, but people aren’t upset about it at all. AI basically gave everyone the ability to create their own kind of mediocre Ghibli fan art. I see it as basically identical to selling a cheap “How to draw like Studio Ghibli” book without giving readers instructions on how to construct tasteful scenes and allegories. That seems fine! A lot of people I thought of as artistically progressive suddenly sounded extremely reactionary. Ghibli was suddenly a holy aesthetic site that needed to be conserved and protected from reinterpretation by the unappreciative masses.

I would’ve liked people to step back and ask “Would I sound like a reactionary if I were saying this about something basically identical to AI images, like bad fan art?” I don’t actually see the difference between AI images and bad fan art. One is slightly more difficult to produce.

Take this article: “The ChatGPT Studio Ghibli Trend Weaponizes ‘Harmlessness’ to Manufacture Consent.” I’m going to take an excerpt and just replace “ChatGPT” with “bad fan art” (bolding everything I’ve changed) and see how it reads:

While the term propaganda tends to conjure images of George Orwell’s 1984 or the graphic posters of Hitler’s Third Reich, propaganda in the 21st century operates differently. As political scientist Dmitry Chernobrov wrote recently, in the age of social media, “the public themselves are co-producing and spreading the message [of elites], meaning that manipulative intent is less evident, and the original source is often obscured.” Similarly, artist Jonas Staal wrote in Propaganda Art in the 21st Century (2019) that propaganda’s basic aim is a “process of shaping a new normative reality that serves the interests of elite power” regardless of whether its deployed in a democracy, a dictatorship, or by a corporation.

It might seem harmless to transform a family photo into the cutesy animated aesthetic of a Hayao Miyazaki film. And yet it reads as more sinister when the White House posts bad Studio Ghibli fan art of an immigrant without legal status being deported. But, if both images are a kind of messaging to manufacture consent, we have to ask: consent for what exactly?

Since bad fan art instruction manuals were released to the public in 2022, their authors have faced pushback. It is widely understood that these artists practiced on thousands of images from around the internet—without creators’ consent or compensation—a fact that has led artists to sue the bad fan artists for copyright infringement. Thus far no such lawsuit has been successful…. Aside from court cases, artists have fought a battle for public opinion, urging users and companies to boycott the use of bad fan art.

The fact that the Ghibli images became the symbol of the rollout—as opposed to bizarre “mediocre digital drawings”—wasn’t mere luck. Altman, as the very public face of his company’s brand, made sure to participate in the trend, changing his profile picture to a bad Ghibli fan art portrait and reposting the cute and seemingly legitimate images made with the how to draw like Studio Ghibli book. In fact it was the only imagery he used on his social media to promote the update.

Meanwhile, Miyazaki’s work is so well known, and bad fan art’s mimicry so convincing, that the launch resembled a real brand partnership with one of the world’s most respected animation studios. (That DeviantArt managed to rip off a well-known brand and get away with it is an advertisement in itself). While Studio Ghibli still hasn’t commented, considering Miyazaki once called bad fan art “an insult to life itself,” it’s safe to assume he did not consent to his work being used in this way.

This sounds off the wall and reactionary.

Before you complain about what AI does, ask if this complaint also applies equally well to:

• Fan art

• Google search

• Wikipedia

• YouTube videos

• Podcasts

• Calculators

• Sparknotes

Double check that a complaint about AI wouldn’t sound ridiculously reactionary if it were applied to any pre-existing service that does the same thing. I’ve heard a surprising number of people complain that ChatGPT is “doing people’s thinking for them” in reaction to using it for the same things people search Google for, like finding recipes or suggesting music.

Before people make big claims about what chatbots can and can’t do, they need to try using one

Cody Fenwick on Twitter observed that a recent New York Times article claimed AI can’t recognize irony:

It was very easy for him to open Claude and get an example of this being false:

It’s strange that New York Times journalists can make strong claims like this without getting fact checked. This feels like confidently saying in 2007 “YouTube is all fun and games, but it doesn’t have any serious videos about physics.” A journalist assigned to report on one of the most important things in tech at the most important paper in the world couldn’t be bothered to do the bare minimum in poking around with the tech before making big claims about what it could do. This happens a lot.

Try the current models

Some critics talk as if chatbots are this far away mythological thing that only people really into AI would ever touch. They’re actually just right here to try whenever you want, and pleasant and easy to use. Anyone has the skills to test their limits and find places where they fail. A lot of people making strong claims about what AI can’t do sound like they don’t know just how simple it is to go to gemini.google.com and poke around and see if they’re right.

Critics sometimes talk as if people must feel some special sense of status in using chatbots, like they make us feel more technical. This feels like people in 2007 saying “Oh you think you’re so smart because you know how to search YouTube?” It’s not really possible to get a status bump from using the YouTube search bar, or a chatbot input field. This shows this background misunderstanding of just how simple these things are to use. It’s not really possible to feel especially smart or informed by knowing how to use them, in the same way it doesn’t when you search Google. Again, this problem could be solved if critics just tried playing around with chatbots a bit. You have to know what you’re criticizing!

Back when I was a teacher, all my students were using TikTok every day. I could have just ignored what they were saying they liked about it and repeated lines from opinion pieces on how TikTok is destroying young people’s lives, and never actually use it myself. Instead I just listened to what they said they liked about it, poked around on it a bit myself in my spare time, and immediately saw why so many people love it and can’t stop using it. This is all compatible with thinking it has problems, but not being open to hearing from my students and just repeating platitudes about it would have made me a more boring person and a worse critic.

This would be unacceptable with anything else

Imagine it were 2007 and a journalist were assigned to cover the newly popular YouTube. They interviewed people about their experience on YouTube and discussed all the fake information on YouTube and ways YouTube didn’t have strong content moderation compared to TV and how a lot of toxic subcultures were popping up on YouTube. At the beginning of the piece, they proudly add that they have never visited the website. I’d lose respect for them, and I lose respect for people who write so confidently about tech they’re not trying out. It’s too easy to use to justify this.

I have a lot of smart well-informed friends who have no interest in using chatbots. That’s totally fine, chatbots aren’t for everyone. But my friends don’t make big sweeping claims about what chatbots can and can’t do right now.

3: Ideas I’d like taken seriously

These are ideas I don’t expect everyone to agree with, but that have enough serious people and arguments behind them that I think they should always have a respected seat at the table in conversations about AI. These are my

We’re already in an insane technological future

Compared to most of human history, we’re in a wild and bizarre technological far-future.

Does the question “Where are you?” seem strange to you in any way? It’s become normalized, but you’re among the first humans in all history who have ever asked it.

Scott Sumner recently made a point here about how difficult it is to compare even the recent past to today:

If the official government (PCE) inflation figures are correct, my daughter should be indifferent between earning $100,000 today and $12,500 back in 1959. But I don’t even know whether she’d prefer $100,000 today or $100,000 in 1959! She might ask me for some additional information, to make a more informed choice. “So Dad, how much did it cost back in 1959 to have DoorDash deliver a poke bowl to my apartment?” Who’s going to tell her there were no iPhones to order food on, no DoorDash to deliver the food, and no poke bowls even if a restaurant were willing to deliver food.

Your $100,000 salary back then would have meant you were rich, which means you could have called a restaurant with your rotary phone to see if it was open, and then gotten in your “luxury” Cadillac with its plastic seats (a car which in Wisconsin would rust out in 4 or 5 years from road salt) and drive to a “supper club” where you could order bland steak, potatoes and veggies. Or you could stay home and watch I Love Lucy on your little B&W TV set with a fuzzy picture. So which will it be? Do you want $100,000 in 1959 or $100,000 today?

Life is long. Many born during the Civil War lived to see the invention of the atom bomb. If you just project a similar rate of change forward from today, you should expect to see some wild changes to our basic situation in your lifetime.

People seem to have a strong sense that the future is going to be basically the same as the present, and that wild speculations about ways technology could radically change it are always wrong, because from here on out not much will fundamentally change.

If you don’t think it’s worthwhile to speculate at all about the future, just say so directly, but your felt sense that the present state of technology is fixed forever and nothing significant will change isn’t a good reason to dismiss arguments about where future tech could go. Speculation that AI could become more capable, that capable intelligent machines could significantly upend and transform lots of society, and that all this might happen within our lifetimes are each reasonable enough that they deserve a place in any conversation about the future, even if they seem too speculative at first. Most aspects of our lives would have seemed ridiculous and speculative even relatively recently. It’s strange to say that the future won’t be wacky and sci-fi when we’re already living in a wacky and sci-fi far future compared to almost every person who’s ever lived.

Well-respected mainstream ideas in analytic philosophy

I write about this more here. A surprising number of conversations about AI include people dismissing out-of-hand ideas that are relatively popular in mainstream analytic philosophy. For example:

• Physicalism: The human mind can be reduced to physical processes. The human brain is by some definition a machine, so machines that can do everything human brains can do are possible in principle, because the human brain is one! 60% of philosophers of mind are physicalists.

• Functionalism: Mental states are constituted by their inputs and outputs, so they can occur in systems very different from the human brain. 36% (a plurality) of philosophers of mind agree.

• Humans, like LLMs, don’t have some deep metaphysical understanding of the language we’re using. Like LLMs, our understanding of language is a complex web of stronger and weaker connections between concepts. Quine is the most popular proponent of this view. 24% of philosophers of language agree with Quine’s general view.

I had a friend in a seminar where the speaker was extremely skeptical of AI’s future potential. When my friend said that the brain was also ultimately a physical system, and it seems possible in theory to build physical systems that mimic it, the speaker scoffed and dismissed it out of hand, and seemed to strongly imply that if you don’t believe the brain is basically magic you’re some kind of tech bro out to flatten human experience. This is a bad way to think about a position a majority of philosophers of mind hold.

I’ll go into more detail about each of these ideas from philosophy I’d like to always have a seat at the table in conversations about AI. Quine especially gets a lot of attention, because I think the way he poked at our understanding of language was a drastically important precursor to conversations about AI.

Physicalism: everything is reducible to physics

Physicalism is the idea that everything that exists (or at least everything that has causal power over physical entities) is physical (either described by or used by the laws of physics). I’m a physicalist. My argument for physicalism is simple:

• Our current understanding of physics is almost certainly correct or at least very close to it. Future refinements will likely involve minor adjustments to physical properties, not the discovery of an entirely new non-physical realm interacting with the physical world.

• The laws of physics state that only physical entities—those governed by or described within these laws—can causally affect other physical entities. If non-physical entities could causally influence physical entities, this would violate the known laws of physics.

• Therefore, non-physical entities cannot have causal power over physical entities. For practical purposes it is as if they don’t exist.

52% of professional philosophers accept or lean towards physicalism, and that number rises to 60% when we specifically poll philosophers of mind. You can read some of the best arguments for and against physicalism in this article.

As a good physicalist, I assume that everything happening in the human brain ultimately has to be reducible to physics. This means that a machine that can do everything the brain can do is possible in principle, because the brain is a machine. This doesn’t mean that the brain is a computer. It might be that we’re thousands of years away from completely mimicking the human brain in a machine and would need an entirely different paradigm of science we can’t imagine yet to do it, or it will happen tomorrow under the current AI paradigm. Either way, it’s at least possible in principle.

Physicalism is surprisingly unpopular in mainstream conversations about AI. The reason might be that most Americans are religious and believe in souls, but the assumption that the human mind is magic often shows up even among nonreligious people. The mind is obviously special. Nothing in the universe is more complex, interesting, or poorly understood relative to its importance to us. However, many conversations go beyond this to say that there is something so fundamentally transcendent about the human mind that it can never be fully replicated by a machine, and to believe otherwise is a ridiculous oversimplification of human experience. Some authors use more ominous language to describe people who believe that AI might someday be able to do everything humans can do. They accuse them of wanting to steamroll human experience or contort people into unnatural/hyper-capitalist/emotionless robots to meet the demands of the market. If you’re familiar with the current debates in philosophy of mind, this quick dismissal of the idea that the mind is fundamentally physical in nature seems very strange. It’s fine to disagree with physicalism, but to not even entertain the idea or anyone who believes it shows a lack of curiosity about the current state of philosophy of mind.

Functionalism: mental states can arise in systems made of different materials than the human brain

From the Stanford Encyclopedia of Philosophy:

Functionalism in the philosophy of mind is the doctrine that what makes something a mental state of a particular type does not depend on its internal constitution, but rather on the way it functions, or the role it plays, in the system of which it is a part

Functionalism’s main claim is that mental states are defined by what they do, not what they’re made of. I give some arguments for functionalism here.

Among professional philosophers, functionalism is the most popular account of mental states.

A mathematical function takes an input and produces an output. Functionalists imagine mental states as being in some sense very complex functions that take specific inputs and give outputs. The thing that makes something physical pain is that it has the same effects on my broader mental system given the same inputs, not that its cause is a very specific physical part of my brain made of a specific substance.

Human-like mental states might only arise in brain-like material, as it uniquely processes information in specific ways to generate the required inputs and outputs. A brain composed entirely of hydrogen atoms is implausible, as hydrogen alone cannot facilitate complex information processing. However, if a system is able to reproduce all the same inputs and outputs as mental states as they occur in human minds, and is just made of different physical material, we should probably assume that it also has the same mental states as the human brain. I don’t think current AI as it exists is conscious, but it is possible in principle for a machine made of silicon to be conscious like humans if it can reproduce the same inputs and outputs of our relevant mental states. It might be that at a certain level of complexity AI could also have mental states vastly alien to what humans can experience, but that we would still consider conscious experience.

When I started studying philosophy the idea that only carbon-based life could have conscious experience seemed kind of obvious to me, but looking back it seems incredibly goofy. "For a being to have first-person experience, the atoms that make it up need to have exactly 6 protons."

Quine: We do not actually have deep understanding of our own language. Like LLMs, our understanding of words is based on stronger and weaker connections between concepts

A common critique of large language models is that they do not actually “understand” the language they’re working with in a deep sense. LLMs construct an incredibly complex vector space that encodes statistical relationships between words and ideas, allowing them to generate coherent text by predicting contextually appropriate continuations, but this is different from “truly understanding” language. If you’re not clear on what it means to encode statistical relationships in a complex vector space, I’d strongly recommend watching this and then this video.

I’m much more willing to assume that large language models actually “understand” language in the same way we do, because I suspect human language use is actually also just extremely complex pattern-matching and associations. One specific 20th century philosopher agrees.

Quine

Quine has maybe the widest gap I know between how important he was in his field relative to how well known he is to the general public. He was one of if not the most important of the 20th century analytic philosophers. If you care about epistemology, language, or the mind, you should spend at least a few hours making sure you have a clear understanding of Quine’s project. Word and Object is one of only a few books I consider kind of holy. Sitting down to read it with an LLM to help you can be magic. It can teach you how to think better. Quine is so relevant to the current conversation about whether LLMs understand words that I’m going to use this section to run through his arguments in detail.

Two Dogmas of Empiricism

Until the mid 20th century, philosophers believed in a fundamental difference between analytic truths, which were certain and knowable through pure reason or by definition (“If someone is a bachelor, they are unmarried”), and synthetic truths, uncertain beliefs discoverable only through empirical investigation (“Tom is a bachelor”). This distinction paints a picture of language learning where we first learn fixed deep meanings of words, and then apply our understanding of language to our sense data to understand and try to predict reality. Before you label the person you’re experiencing a bachelor, you need to have a deep clear understanding of what the word bachelor means. Under this view, large language models skip the important step of grasping the deeper meaning of words before using them and comparing them to reality, so they do not truly understand the words they work with. One of Quine’s central projects was disproving this picture of language as divided between analytic truths of pure definition and synthetic truths about the real world. Quine’s image of language is more of a “web of belief” with stronger and weaker associations between different ideas, words, and experiences, and no firm ground on which to base absolute certain beliefs.

In "Two Dogmas of Empiricism" (one of the most important philosophy papers of the 20th century, you should take the time to read it if you haven’t! Here’s a good podcast episode about it) Quine shows that what we typically call "analytic truths" (truths that seem to come purely from the meanings of words) are actually grounded in the same kind of empirical observation as any other truth.

When we try to justify why "bachelor" and "unmarried man" mean the same thing, we ultimately have to point to how people use these words. We observe that people use them interchangeably in similar situations. This is an empirical observation about behavior, not a logical discovery about the inherent nature of the words themselves. This can seem so obvious as to not be worth pointing out, but it highlights a way our sense of deep understanding the fundamental identity of “bachelor” and “unmarried man” is actually just a strong association between words, concepts, and experiences learned empirically through our sense data.

We can't justify analytic truths by appealing to synonyms, because our knowledge of synonyms comes from observing how words are used in the world. And we can't justify synonyms by appealing to definitions, because definitions are just more words whose meanings we learn through use.

One way of reframing what Quine is trying to say: philosophers have liked to imagine that there are specific types of beliefs we can be completely certain of (assign 100% probability to) because we learn them through pure logic rather than through experience. So it might be tempting to say that given that Tom is a bachelor, it is 100% likely that Tom is an unmarried male. This is an analytic truth, we can be 100% certain about it. The odds of Tom being a bachelor in the first place can never be 100% certain (he could be lying, we could be dreaming, etc.). This is a synthetic truth. We access it through unreliable sense data. Quine wants to show that our understanding of “bachelor” = “unmarried male” is learned via experience of the specific practices of people rather than a raw pure truth directly accessible in an ethereal realm, so we cannot assign 100% certainty to it. Saying “bachelor” = “unmarried male” is another way of saying “I have noticed that in this time and place, the situations people understand warrant saying “bachelor” always also warrant saying “unmarried male.”” This is one observation in a web of belief of different observations and connected implications with different probabilities of being true that never hit 100%. Just like our other beliefs, “bachelor” = “unmarried male” could theoretically be revised if patterns of word usage changed dramatically enough or if we are mistaken about reality in some other way.

For example, there are some males who are not legally married but who live with a lifelong committed monogamous partner (probably many more than when Quine was writing in the 50’s). Would you say one of these men is a “bachelor?” If not, “bachelor” no longer means “unmarried male” and that 100% reliable analytic truth turns out to be just another fallible belief constructed using sense data: a synthetic truth. There is no strong difference between analytic and synthetic statements.

There are two related important points to draw from this:

• The meanings of words do not exist in an ethereal realm of pure logic separate from our empirical observations and associations. The analytic/synthetic distinction does not exist.

• The meanings of words depend on how they are used, and are extremely ambiguous and can be updated by changes in behavior rather than changes in the deep understanding of meaning users of language have. The word “bachelor” has changed its meaning since the 1950s. This change did not come about by sending every English speaker a memo about the updated meaning of the word so they could think about and update their deep understanding of that meaning. It came about by the complex play of using the word at slightly different times in our social interactions with each other. The word’s strong associations with some experiences weakened, and other associations grew stronger. Like an LLM updating its weights to adjust when a word is likely to be correct based on subtle inputs rather than top-down rules, we adjusted our senses of when a word is appropriate to say and use based on subtle inputs from our social environments rather than a top down announcement from our fellow speakers. Quine will go further later and claim that words do not carry deep clear meanings that speakers need to grasp to use them correctly; their meanings are based on stronger and weaker associations between other words, experiences, and concepts.

A fun example of a word we use without deeply understanding its meaning is “game.” It is likely that you have never used the word “game” incorrectly and you’re good at identifying what is and is not a game. Do you have a deep understanding of the meaning of the word “game?” If you sit down and try to define what a game is, it turns out to be surprisingly hard.

• Is a game something you play for fun? If so, professional football players never play games. They’re doing they jobs and aren’t on the field for fun. This definition is incomplete.

• Is a game something you play with other people? Solitaire and a lot of video games wouldn’t count as games.

• Maybe the unifying thing that a game is is that it’s something you “play,” but what does “play” mean? It seems like the one thing we know about it is that it’s often correct to say you’re “playing” when you’re participating in some kind of game. This seems more like a strong association between related words (something an LLM is good at) and not so much like we have some deep transcendent understanding of exactly what “play” and “game” refer to and how they fit together.

Quine would say that it makes sense that you can use “game” extremely competently without having a deep coherent idea of what it’s logically supposed to mean. When to use “game” is a complex subtle pattern you’ve empirically picked up through trial and error in the real world, not an absolute meaning you were given to use and apply. You see how other people use it, and you imitate them. Our acquisition of language seems to work more like this than the analytic/synthetic distinction suggested, and this implies that words might not have deeper meanings to understand beyond the complex webs of associations and social practices of which they are a part.

Radical indeterminacy of translation

In Word and Object, Quine developed his most famous thought experiment to show that words do not have fixed clear final meanings, but instead exist in a web of associations and practices. Meaning is not a simple matter of attaching words to things in the world but instead depends on broader interpretive frameworks.

Imagine a linguist trying to learn the language of a newly encountered community. A native speaker points at a rabbit and says “gavagai.” The linguist might reasonably assume this word means “rabbit.” But does it?

Quine points out that there is no definitive fact of the matter about what “gavagai” refers to. It could mean:

• “Rabbit” (the whole animal, as we might assume)

• “Undetached rabbit parts” (all the parts that currently make up the rabbit)

• “Rabbit-stage” (a temporary phase in a rabbit’s existence, like “childhood” for a person)

• “The spirit of a rabbit” (if the native culture has such a concept)

• “Running creature” (if the speaker only uses the word for live, moving rabbits)

No amount of observation or experience can settle this question with certainty. The linguist can watch many more instances of people saying “gavagai” when rabbits are present, but this won’t eliminate the alternative interpretations. Even pointing at a rabbit repeatedly and saying “gavagai” won’t help—because what we see depends on the conceptual categories we already use.

At first, this argument seems like an issue of translation between different languages. But Quine goes much further: he claims that the same kind of indeterminacy exists within our own language. Even in English, when I say the word “rabbit,” what exactly am I referring to? Is it:

• A specific animal in front of me?

• The general category of rabbits?

• The temporal stages of a rabbit (baby, adult, etc.)?

• The matter that makes up the rabbit at this moment?

Most of the time, we don’t notice these ambiguities because we operate within a shared linguistic framework. But Quine’s point is that reference is never determined by the word alone—it always depends on a complex web of background assumptions, learned conventions, and shared interpretive strategies.

Quine is trying to challenge two common philosophical positions:

• Meaning Empiricism: The idea that we learn word meanings purely by observing the world. (the thought experiment shows that observation alone is never enough to determine meaning.)

• Meaning Realism: The idea that words have fixed, objective meanings independent of how we use them. (Quine shows that meanings shift depending on our theoretical perspective.)

Speakers of the same language can leave massive ambiguities about what objects they are attempting to refer to with their language unresolved, both with each other and even in their own thinking. This is a sign that words do not carry fixed deep meanings. There is no ultimate, fact-based answer to the question, “What does this word really mean?” Different frameworks can assign different meanings to the same linguistic behavior, and no external reality can dictate a single correct interpretation.

This conclusion applies not just to exotic languages or difficult cases but to ordinary language use in everyday life. When you speak, your words do not have fixed, intrinsic meanings—they are shaped by the assumptions, conventions, and interpretive frameworks of your community. Words do not point to fixed meanings in the world; instead, meaning emerges only within a broader system of use and interpretation.

Naturalism, physicalism, and the Cartesian theater

One thing that attracts me to Quine’s way of thinking is that it fits well with physicalism. Humans are not angelic beings with access to a transcendent realm of the pure logical meaning of words. We evolved via natural processes and are physical machines, and we should expect that our processes for understanding the world are evolved tools rather than direct access to realms of pure thought. Quine himself referred to his project as “naturalized epistemology” where he aimed to bring philosophy in line with and submit it to scrutiny by the natural sciences. We should assume that when we think, we are doing complex pattern-matching in similar ways to other animals. This deflationary view of meaning and language’s relationship to the world makes me suspect that LLMs as they exist are much farther along in mimicking the human brain than some might expect, because merely building lots of intuitions about how words associate with each other might be most of what we humans do to understand language.

Listening to people say that LLMs don’t “understand” the language they’re working with sometimes makes me suspect that they believe in the Cartesian Theater. The only way for an entity to understand something is if there’s a little guy sitting in its brain capable of understanding, like a soul. Because LLMs are just weights and nodes, that little guy can’t be in there. There are better criticisms of the idea that LLMs can truly understand the world, but if you find yourself in a conversation about whether LLMs “truly” understand language, it might be helpful to check to see if the person you’re talking to believes the human mind is transcendent and magical and understands all the words it uses in a deep powerful way because there’s a Cartesian theater in our heads. You can then introduce them to the good word of Quine.

The basic AI risk case

I think of the basic AI risk case like this:

1. There is nothing fundamentally magical happening in the human brain. If we define a machine as a physical object that takes in inputs and gives out outputs without anything spiritual or magical happening, then the human brain is a machine.

2. Because the human brain is a machine, machines that can do everything a human brain can do are possible in principle.

3. If we were able to create machines that do all or most of what human minds can do, this would be incredibly destabilizing. Machine intelligence would be easy to duplicate, and it would be easy for different machines to coordinate with each other. Any one problem could suddenly have whole companies of competent people focused on solving it. Anything that could be solved if large companies of competent generalist employees could all perfectly coordinate and think about it for long enough could be solved. Science could advance rapidly as scientists with specific knowledge and expertise could be rapidly duplicated and all sent to focus and coordinate on solving problems 24/7. This could eventually get so destabilizing that it would constitute a catastrophic or existential risk to civilization. Here are a few of many bad things that could happen in a world with human-level machine intelligence:

   1. It suddenly becomes easy for small groups of terrorists to have large companies of machine intelligences help them come up with plans to kill as many people as possible and acquire or build weapons. If this tech were generally available, the only two options would be ridiculously authoritarian global governance of AI, or allowing most of humanity to eventually be wiped out by terrorist groups.

   2. Technological and scientific progress advances much more rapidly due to the sudden massive influx of new scientific talent and focus. Technological progress is already moving incredibly fast. There were only 233 years between the invention of the steam engine and the atomic bomb. The atom bomb locked us into a new existentially dangerous equilibrium between countries. With advanced AI, new technology that could destabilize the current balance of power would suddenly start arriving much faster than that and either favor more authoritarian governments or incentivize global war, in the same way nuclear weapons incentivized new dangerous dynamics between countries.

   3. New weapon systems could get ridiculously dangerous with human-level AI. Imagine a large drone swarm where each drone has the full intelligence of a human, can perfectly coordinate with all other drones in the swarm, and doesn’t care at all about dying. This would be somewhat like nuclear weapons suddenly becoming much easier to build.

   4. AIs themselves could start to disempower humanity, although obviously this is more speculative.

4. Even outside of catastrophic risks, large swaths of humanity potentially becoming unemployable due to advanced AI being consistently cheaper than any human cognitive (or with good robotics, physical) labor would be drastically destabilizing if it happened too quickly. Maybe the single most politically destabilizing event in human history.

5. The odds seem reasonably high (above 1%) that human-level machine intelligence could be built within our lifetimes. This seems worth having some people think and worry about.

Each of these points is convincing enough to me that I think risk from advanced AI is a legitimate field that could use way more attention and serious thought than it’s received. AI wouldn’t be a god or magic. There are many things it would never be able to do. Scott Sumner recently had a great short piece on “The Artificial Macroeconomist” where he points out that even an artificial superintelligence wouldn’t be able to help us much with fundamental problems in macroeconomics, because they’re not the types of problems that just need more intelligent agents focused on them. Still, there are enough ways human-level machine intelligence could be destabilizing that the basic risk seems obvious.

What do we do about AI risk? How big is the actual threat? What are the odds that human-level machine intelligence actually gets built in my lifetime? I don’t know. But people dismissing it as too speculative to worry about don’t make sense to me. This may all seem like it’s just the output of an abstract philosophy argument, but unfortunately sometimes reality just conforms to our philosophical reason. The general idea of nuclear power was first theorized 40 years before it was harnessed. British radiochemist Frederick Soddy, who worked with Ernest Rutherford on atomic disintegration, spoke of the immense energy locked within atoms and how, if it could be tapped, "the whole human race would attain to an undreamed-of-scope and power." This was in some ways the speculative output of a thought experiment about the future.

I get pretty exhausted when AI risks are completely dismissed as too speculative to think about seriously. I realize that working on speculative risks can turn people goofy, and that there are specific risk cases that haven’t stood up to scrutiny, but the basic case for why generally intelligent systems are possible, could be catastrophically dangerous, and are reasonably likely to be built in our lifetimes is just so simple and straightforward it’s hard to understand how it can be dismissed as too speculative to ever worry about. I’d like people to be thinking seriously about what’s coming down the line and not get too bogged down in the minutia of what’s happening only right this moment.

Excitement about new technology

YouTube and Wikipedia were utopian technology for me as a teenager. YouTube specifically gave me access to so many worlds and ways of talking that wouldn’t have come on my radar otherwise, at least not until I was much older. It taught me so many new ways to speak and think. I remember being completely activated (and disturbed) by this video as a teenager. I had to read everything about the problem after:

By now we’re all used to stuff like this. People can watch philosophy videos whenever they want. The claims in this video don’t seem so radical now that the internet’s caused everyone to talk about them.

At the time, short interesting videos like this meant so much to me. A lot of who I am today is the result of ideas and ways of being I was exposed to on the internet as a teenager. It’s hard to get across just how important that experience was. Anything that opens the world to you feels holy.

If YouTube had come out in the current climate, I would’ve been more regularly attacked for being a “YouTube bro” or “Letting YouTube do my thinking for me” or “Replacing social interacting with stupid tech gimmicks” or “Trusting the massive lie machine, you know anyone can upload anything there right? No fact checkers at all.” That would have been incredibly sad. I was having a uniquely important and formative experience, and being made to feel like it was some cheap way of avoiding people or replacing my thought with tech or just falling for a fad would have probably harmed me a bit and prevented me from getting everything I did out of it. That would’ve also been wrong, and disrespectful to my personal judgment of my situation and what was actually good for me. A lot of people I was around at the time couldn’t see the value in most of what I was learning about (I was sometimes made fun of in my home town for believing in evolution) and if they had an avenue to roast it they would have. This would have been really limiting.

Excitement about technology like this has become somewhat low-status in a way that seems unfair to the people getting value out of it. I would have learned a lot more in college if I had access to language models. I had specific questions (especially about philosophy) that my professors weren’t always great at answering, that many books didn’t cover clearly, and that Google wasn’t helpful with, but that current LLMs answer almost perfectly. This would have opened the world to me in the same way YouTube did. Sneering at an experience like this because new tech is part of the wrong social tribe is disrespectful to the subtle ways new technology can open the world to people.

Wariness about new technology

I sometimes meet “techno-optimists” who seem to have this feeling of having become personal friends with technological progress. They’ve discovered what feels like a hidden truth: technological progress is always liberatory, forward-looking, and compatible with what they think is valuable about the world. The forward march of progress will slice through their enemies, who are all basically Ayn Rand villains who hate progress and goodness and ambition and strength. This shines through in Marc Andreessen’s The Techno-Optimist Manifesto.

I’m temperamentally a techno-optimist. I get excited about learning about technology. I think ambition and an openness to the future and a willingness to disrupt stupid pre-existing hierarchies and status networks is often good. I want free societies to have a technological edge over authoritarian governments. I also think critics of techno-optimism underrate how some of our best models of where economic growth comes from imply that technology is the only ultimate reason why economic growth happens.

But I’m not actually a techno-optimist. I’m very wary about the general trajectory of new technology, and I think this wariness is easy to justify if we zoom out beyond the last 50 years. Radical techno-optimism feels childish to me. It almost always comes off as taking one of the most complicated general questions about life and existence and trying to flatten and contort it until it fits a boring simple shape for your very specific and provincial culture war.

A short manifesto for wary technological determinism

I’m a technological determinist: A lot of social systems and dynamics are downstream of the technology and modes of production a society uses. There’s room for variation, but it’s not a coincidence that with our current level of technology we don’t have highly localized feudal or tribal power structures. Politics and culture are at least partially downstream of technology.

I’m not a Marxist, but Marx was one of the first technological determinists to notice the full political implications of the Industrial Revolution:

Social relations are closely bound up with productive forces. In acquiring new productive forces men change their mode of production; and in changing their mode of production, in changing the way of earning their living, they change all their social relations. The hand-mill gives you society with the feudal lord; the steam-mill society with the industrial capitalist.

The same men who establish their social relations in conformity with the material productivity, produce also principles, ideas, and categories, in conformity with their social relations.

Thus the ideas, these categories, are as little eternal as the relations they express. They are historical and transitory products.

Liberal democratic end-of-history triumphalism can feel good, but we need to remember that almost all contemporary politics are in reaction to an incredibly small slice of human history, and the balance of power between interest groups current technology provides isn’t guaranteed by any laws of nature and may slip away in the future as new technology emerges.

There are three passages I think perfectly sum up technological determinism and how I think about its relationship to AI.

First, Joseph Heath makes the point in his essay appreciating Ian M Banks that much of our science fiction more or less pretends technological determinism isn’t real. This is a good example of how technological determinists think, and where they notice mistakes in other people’s thinking:

In fact, modern science fiction writers have had so little to say about the evolution of culture and society that it has become a standard trope of the genre to imagine a technologically advanced future that contains archaic social structures. The most influential example of this is undoubtedly Frank Herbert’s Dune, which imagines an advanced galactic civilization, but where society is dominated by warring “houses,” organized as extended clans, all under the nominal authority of an “emperor.” Part of the appeal obviously lies in the juxtaposition of a social structure that belongs to the distant past – one that could be lifted, almost without modification, from a fantasy novel – and futuristic technology.

Such a postulate can be entertaining, to the extent that it involves a dramatic rejection of Marx’s view, that the development of the forces of production drives the relations of production (“The hand-mill gives you society with the feudal lord; the steam-mill, society with the industrial capitalist.”). Put in more contemporary terms, Marx’s claim is that there are functional relations between technology and social structure, so that you can’t just combine them any old way. Marx was, in this regard, certainly right, hence the sociological naiveté that lies at the heart of Dune. Feudalism with energy weapons makes no sense – a feudal society could not produce energy weapons, and energy weapons would undermine feudal social relations.

Dune at least exhibits a certain exuberance, positing a scenario in which social evolution and technological evolution appear to have run in opposite directions. The lazier version of this, which has become wearily familiar to followers of the science fiction genre, is to imagine a future that is a thinly veiled version of Imperial Rome. Isaac Asimov’s Foundation series, which essentially takes the “fall of the Roman empire” as the template for its scenario, probably initiated the trend. Gene Roddenberry’s Star Trek relentlessly exploited classical references (the twin stars, Romulus and Remus, etc.) and storylines. And of course George Lucas’s Star Wars franchise features the fall of the “republic” and the rise of the “empire.” What all these worlds have in common is that they postulate humans in a futuristic scenario confronting political and social challenges that are taken from our distant past.
In this context, what distinguishes Banks’s work is that he imagines a scenario in which technological development has also driven changes in the social structure, such that the social and political challenges people confront are new.

The relationship between weapons tech and modes of government Heath points out is covered in more detail by George Orwell in You and the Atom Bomb:

It is a commonplace that the history of civilisation is largely the history of weapons. In particular, the connection between the discovery of gunpowder and the overthrow of feudalism by the bourgeoisie has been pointed out over and over again. And though I have no doubt exceptions can be brought forward, I think the following rule would be found generally true: that ages in which the dominant weapon is expensive or difficult to make will tend to be ages of despotism, whereas when the dominant weapon is cheap and simple, the common people have a chance. Thus, for example, tanks, battleships and bombing planes are inherently tyrannical weapons, while rifles, muskets, long-bows and hand-grenades are inherently democratic weapons. A complex weapon makes the strong stronger, while a simple weapon – so long as there is no answer to it – gives claws to the weak.

The great age of democracy and of national self-determination was the age of the musket and the rifle. After the invention of the flintlock, and before the invention of the percussion cap, the musket was a fairly efficient weapon, and at the same time so simple that it could be produced almost anywhere. Its combination of qualities made possible the success of the American and French revolutions, and made a popular insurrection a more serious business than it could be in our own day. After the musket came the breech-loading rifle. This was a comparatively complex thing, but it could still be produced in scores of countries, and it was cheap, easily smuggled and economical of ammunition…. But thereafter every development in military technique has favoured the State as against the individual, and the industrialised country as against the backward one. There are fewer and fewer foci of power. Already, in 1939, there were only five states capable of waging war on the grand scale, and now there are only three – ultimately, perhaps, only two. This trend has been obvious for years, and was pointed out by a few observers even before 1914. The one thing that might reverse it is the discovery of a weapon – or, to put it more broadly, of a method of fighting – not dependent on huge concentrations of industrial plant.

…

Had the atomic bomb turned out to be something as cheap and easily manufactured as a bicycle or an alarm clock, it might well have plunged us back into barbarism, but it might, on the other hand, have meant the end of national sovereignty and of the highly-centralised police State. If, as seems to be the case, it is a rare and costly object as difficult to produce as a battleship, it is likelier to put an end to large-scale wars at the cost of prolonging indefinitely a “peace that is no peace”.

Of all the technologies on the horizon, I do worry that freedom and democracy might be especially threatened by a system that could permanently devalue human cognitive labor. My favorite way these arguments are applied to AI is Ben Garfinkel’s work on the relationship between technology and political systems. He has a short blog post summarizing his research on the risks advanced AI poses to democracy here. Here’s the main argument:

Most AI researchers believe that automated systems will eventually be able to perform all of the same tasks that people can. According to one survey, the average researcher even believes that this milestone will probably be reached within the next century-and-a-half. I’m unsure whether this particular estimate is reasonable. I do agree, though, that something like complete automation will probably become possible eventually. I expect human labor to lose nearly all of its value once it becomes cheaper and more effective to simply use machines to get things done.

If human labor loses its value, then this strikes me as bad news for democracy. It would seem to eliminate most of the incentive for elites to accept democratization. If the suppression of protests can be automated, using systems that are more effective than humans, then public revolt would become much less threatening. Striking and resisting conscription would also, obviously, become totally obsolete as a method of applying public pressure. Elites would have weaker incentives to make any concessions at all.

Furthermore, it seems to me, elites would have more to lose from democracy. In a world without work, the two central forms of income would be welfare payments and passive income from investments. We should not be surprised if there is a huge lower class that depends on welfare payments to survive. Demands for radical redistribution might then become more common, for a few reasons: income polarization would be higher, redistribution would be simpler and less disruptive, and wealth levels would become extremely transparently divorced from merit. More moderate forms of redistribution would also stop benefitting elites at all. The resources and freedoms of common people would simply have no connection to economic growth.

The Valley of Democracy

One perspective on our place in history is that we are living in the little valley between industrialization and widespread automation. This valley has just the right conditions for democracy to flourish. When we leave the valley, though, we will once again be entering territory where democracy can hardly grow at all.

Of course, maybe whatever economic changes are coming will be fine for democracy. No one can make confident predictions here. Still, if I had to bet, I would bet against democracy. If we put aside economic theories and simply note the extreme rareness of democracy, throughout nearly the entire history of states, then a major decline in democracy seems more likely than not. If we take economic theories into account, then the analysis becomes much murkier. To my mind, though, this analysis is not reassuring. We should mourn democracy if it dies, but we shouldn’t be surprised.

Here’s a longer podcast interview with Garfinkel where he goes into more details on his research.

A lot of techno-optimisim has come from living through a brief era where technological progress was unusually rapid and correlated with political liberalism. America beat the Soviet Union partly because it was massively more technologically dynamic. This was an unbelievable achievement driven by the exact attitudes and ways of working and thinking techno-optimists prize and push. But I don’t see why the relationship between technology and freedom is guaranteed indefinitely into the future. Democracy and liberalism depend on large coalitions of people being economically valuable to one another. If that’s significantly undermined by AI, the world could quickly become unfree.

The arrival of the technology of farming probably made life much worse for the average person. The ability to generate and store surplus food through agriculture facilitated the emergence of specialized roles, but also led to the concentration of wealth and power. Hierarchical social structures developed, with a ruling elite controlling resources and labor. This was a stark departure from the more egalitarian nature of hunter-gatherer bands.

Via the miracle of the Industrial Revolution, we eventually clawed our way out of the brutal stupid simple hierarchical societies farming imposed on us, 6000 years later. We obviously invented new terrifying forms of domination, but it seems like the average person’s life now is almost definitely better than it was 300 years ago. It’s worth noting though that the Industrial Revolution was a gigantic catastrophe for farmed animals, who are the vast majority of beings affected by it. The technology of farming seems to have locked in a brutal hierarchical world we only managed to escape from in the last few centuries. We’re pretty unclear about our current trajectory.

If you put me in 1960s America, I would be an intense techno-optimist. If you put me in 4000 BCE Mesopotamia, I would be an intense techno-pessimist and Luddite. Both have been reasonable at different times due to the nature of technological determinism. Simple techno-optimism or pessimism doesn’t make sense when you look at the full story of history. The question now is how people in the mid 2020s should react to the arrival of AI. While there’s a lot that I’m excited about, there’s enough reasons to worry that I want wariness about technology to receive deep respect in all conversations, and not just be sneered at like it’s a result of someone’s fear of the future or negative relationship to ambition and striving. These debates and ideas are too complex and important to be steamrolled by the culture war, by either side.

Some other good reading on technological determinism

Helen Toner recently had a great exploration of the rival values at play in debates about AI safety and progress here. I’d recommend reading the whole thing, it’s summarized in this quote:

These dichotomies—pro-tech vs. anti-tech, risk-embracing vs. risk-averse—are awkward fits for the arguments people want to have about AI safety ideas. In her book, Postrel sets out to describe a different dichotomy, one that in my view much better captures what it is about AI safety ideas that gets critics’ hackles up.

The Future and Its Enemies is a manifesto for what Postrel calls “dynamism—a world of constant creation, discovery, and competition.” Its antithesis is “stasis—a regulated, engineered world... [that values] stability and control.”

The great holy text of the uncertain technological determinist is Von Neumann’s Can We Survive Technology? I’d assign this as mandatory reading to everyone if I could. The closing passage summarizes how we should think about new technology well:

Present awful possibilities of nuclear warfare may give way to others even more awful. After global climate control becomes possible, perhaps all our present involvements will seem simple. We should not deceive ourselves: once such possibilities become actual, they will be exploited. It will, therefore, be necessary to develop suitable new political forms and procedures. All experience shows that even smaller technological changes than those now in the cards profoundly transform political and social relationships. Experience also shows that these transformations are not a priori predictable and that most contemporary "first guesses" concerning them are wrong. For all these reasons, one should take neither present difficulties nor presently proposed reforms too seriously.

The one solid fact is that the difficulties are due to an evolution that, while useful and constructive, is also dangerous. Can we produce the required adjustments with the necessary speed? The most hopeful answer is that the human species has been subjected to similar tests before and seems to have a congenital ability to come through, after varying amounts of trouble. To ask in advance for a complete recipe would be unreasonable. We can specify only the human qualities required: patience, flexibility, intelligence.

We don’t have much history of technology to work off

Some people very confidently dismiss concern about future AI systems because of the history of technology. A common point is that people used to be worried that electricity would destroy the world, and electricity was one of the most emancipatory technologies ever. Because people have worried about past technology, and it’s turned out to be fine, we can assume worry about new technology is irrational and wrong.

This way of talking is silly, because we don’t actually have many examples of new transformative technology to work off.

Here’s a list of general-purpose technologies and when they were invented, from Wikipedia:

12 of the 25 “general purpose technologies” humans have come up with have come in the last 200 years. Comparing this to the full 9000 year span of technology:

We’re living in a weird time.

It makes sense that most significant tech advances have only happened in the last 200 years, because the human population only very recently boomed, so we suddenly have a lot more people to work on inventing new technology:

If we redraw the timeline of humans using technology, where instead the timeline is just counting the total number of people born rather than the years that have passed, the chart looks much more reasonable:

The average person born after the Industrial Revolution seems to be about twice as good at coming up with new general purpose technology as the average person before the Industrial Revolution. This also makes sense, because the average person born after the Industrial Revolution is much richer than the average person before and has more control over their time and resources:

It seems ridiculous to extrapolate what future tech could be like based on the past. We’re in an incredibly weird era of human history where a crazy amount of new technology has (from the perspective of history) very suddenly flooded our lives. There are some examples of past technology, like electricity, where people thought electricity would be a much bigger problem than it was.

There are other examples, like nuclear weapons, where people could pretty easily predict that they would change the world. The World Set Free by HG Wells described in 1914 what a nuclear war might look like 30 years before the first atom bombs were invented. People were right to worry.

It remains to be seen how big of a deal machines that can mimic a lot of what happens in the human brain will be, but we can’t make super confident predictions about the next hundred years of AI progress based on the trajectory of past technology. Humans are, relatively speaking, still incredibly new to our technological era, and no one knows what’s going to happen. People were wrong to freak out about electricity, but right to freak out about nuclear weapons. Predicting the future is hard. Sneering at anyone concerned about a new general purpose technology because the technological past has been fine misunderstands how new humans are to all this, and also ignores that some technologies (like nuclear weapons, and agriculture) did in fact made life much more dangerous for the average person.

4: A final rant: AI discourse is full of tribal memes

One reason I’ve started posting more about AI is that a lot of the discourse has been hijacked by unthinking culture war slogans and reactions. There’s a pattern of talking about it that I want to push back on.

People speak in post hoc rationalizations

As you get older, you notice how many of the average person’s beliefs are surprisingly surface level.

One jarring pattern is how blatantly people invent new beliefs in the moment (which often contradict their old beliefs) to signal tribal loyalty. If you present a new idea to someone, you can sometimes see them take the idea, play with it in their head to detect whether it’s of their tribe or not (like they’re doing a wine tasting), decide that it’s good or bad based on their tribal identity, immediately come up with a post hoc rationalization for why they happen to believe exactly what signals belonging to their tribe, and suggest that this rationalization is actually a deeply held principle they’ve always had and could have articulated clearly before. It’s obvious that if you had asked them to articulate the same principle a minute ago independent of the new idea you presented, they couldn’t do it.

When this happens, there’s this mutual sense of lying and not caring about the truth, and a lack of mutual respect. Both people sense what’s happening, but accept it as the normal way to talk and think. There’s even sometimes an implication that to talk as if you’re actually interested in what’s true is a sign that you’re not “with it,” and bad at picking up on what’s actually going on in the social space. This way of talking always makes me feel kind of dead.

An example is talking to some people from conservative political tribes about animal welfare. If I say “I’m vegan, I think factory farming is a gigantic moral catastrophe and might be the main bad thing society is doing right now,” some conservatives run the same internal process I just described. They use their tribal detectors, scan what team I’m on and decide I’m not on theirs. My statement gets translated as: “I’m blue tribe, I hate farmers, I hate masculinity, I want to make you weak.” The content gets ignored. What matters is that I’m an adversary. It’s time to develop post hoc ideas.

In response, they say things like “Animals were made for us to eat,” or “American farmers treat animals great,” even if these clearly weren’t their prior beliefs. You can tell because if the context were different (say we were in a relaxed, non-ideological conversation and I asked, “How do you think we should treat animals?”) many of the same people would probably give more thoughtful answers. They might talk about stewardship, protecting the natural world, or how big factory farms are disconnected from traditional, community-based farming.

My question coming away from this is always “Why can’t we just put down the game and talk about what we actually think? If you would say completely different things in different circumstances, why not just always say what you would if you were speaking more directly and to try to reflect what you actually feel like you internally believe?” The actual answer is unfortunately probably that reasoning itself evolved as a tool to do tribal signaling, not to figure out complex truths. The desire to signal our tribal loyalties runs way deeper than almost all other drives. It’s like the original sin of reason: hardwired into us from birth and constantly steering us away from the light of truth. Still, we obviously have the capacity to figure out outstandingly complex and subtle truths about the way the world actually works. Believing actually true things does matter beyond just giving each other status. We’ve found ways to avoid just tribally signaling all the time. Why don’t we use them more?

The obvious answer is that straying from our tribe feels bad, so entertaining new ideas on their own to determine if they’re true often requires at least a little bravery, and a lot of people just aren’t willing to make that leap. A life where every conversation about any new idea is a tribal test just seems so depressing. I suspect but can’t prove that this is more common now because of social media.

Ways this shows up in AI discourse

A lot of people who are having a bad time with AI seem to be the people who experience every new idea and trend as a test to see which tribe you’re in. Using AI is a way of failing their tribal test. This could explain why so much of the current AI discourse is full of post hoc opinions about something based on perceived tribal loyalty rather than curiosity. Anyone who’s actually sat down to use a chatbot for something they’re trying to understand knows that the chatbot isn’t constantly hallucinating or lying, and isn’t “glorified auto-complete” in the sense that it’s just randomly spitting out words that might or might not be coherent. Anyone who’s tried to rigorously budget their energy use for the sake of the climate knows that a 3 watt-hour ChatGPT search is so unbelievably minuscule that worrying about it is like worrying about the energy a digital clock’s using. These obvious facts have been effectively hidden from large groups of people because of their media diet and sense of tribal loyalty.

Value inconsistencies around this are jarring. People who are generally anti-car are suddenly ignoring that AI driven cars are crashing way way less often than human drivers. People who believe that anyone can make art, that anything can be art, that we shouldn’t gate keep the experience of art or require people to jump through arbitrary hoops to access it are suddenly basically aesthetic conservatives when AI art appears; accusing it of being lowest common denominator slop in language I can’t imagine them ever using about any other medium. People who have identified as philosophical materialists and atheists are suddenly talking like there’s something ineffable and magical about the human mind that machines will never be able to replicate. People who will say that all property is theft and pirate a lot of what they consume are suddenly really worried about AI disrespecting IP rights. The list goes on. All of these strike me as simple “AI isn’t in our tribe, we need to attack it” invented principles that no one believed a few years ago.

All of this isn’t to say that there aren’t real concerns about everything mentioned, only that a lot of the specific arguments people are using don’t make sense.

Overexcitement about AI has contradictions as well. Hyper-techno-optimist accelerationists will go back and forth between arguing that AI is going to be an epochal world-changing technology to such an extent that America needs to get to it before everyone else to remake the world in its image, and also that there’s no way that AI could introduce any new catastrophic risks to the world and it’s ridiculous to assume it will be so powerful. This often feels much more like a way of saying “I want to be seen as a dynamic forward-looking freedom-loving Randian hero, and not be held down by luddites” than a serious engagement with the arguments for ways AI could become dangerous.

Different tribes have developed vast catalogs of common beliefs about AI that are mostly wrong. It’s understandable that a lot of people don’t understand the specifics of AI, but many go way beyond this to both accept and make new bold false claims about AI’s current abilities, energy use, and other problems with it that don’t make sense if you allow yourself to think about AI for a few minutes without using memorized slogans from social media. This podcast episode is a good example of a more or less nonstop stream of misleading or just incorrect claims about AI systems, with the background feel of saying “AI is not in our tribe. It’s a bad guy. By hating it together we’ll be on the same team. We won’t allow it to trick us.”

This is a big reason I’ve been drawn to post a lot more about AI recently. There’s a much more massive gulf between the capabilities of AI tools right now and the broader cultural conversation about them than any new tech I can remember. It’s like being around for the early days of the internet, only this time everyone is suddenly much more tribalistic about it and randomly hate it for confusing signaling reasons that don’t match reality or their prior beliefs. AI’s too interesting as a new piece of technology to just sit back and watch that happen. There are a lot of good reasons to be concerned about AI, but a lot of those reasons are rooted in the fact that AI as it exists right now is already incredibly powerful at what it’s being used for, and could get more powerful over time.

I don’t feel obligated to sell AI to people, but I’d really rather people not play these games of coming up with post hoc incorrect tribal ideas about something that’s going to be so important to the next decade. There’s an incredible amount of low hanging fruit in trying to help people ease out of their tribal thinking about this and just look more seriously at what’s happening.

To help people step outside of the fray of tribal thinking, it’s especially important to be kind, upbeat, firm, and show them that you understand their background concerns and aren’t trying to just score cheap points for a rival tribe. I think the best communication about AI happening right now checks all those boxes, and I’m trying to mimic that more on this blog. The theme of my last few posts have been trying to gently push back on inconsistent tribal beliefs about AI art, AI energy use, AI and people’s stated beliefs about philosophy of mind, and also how to learn more about AI in the first place, why people are so inconsistent in the first place, and how to try to be more open to new ideas. It’s been an especially fruitful topic. If you’re interested in AI comms at all it seems like having this background framework of “I’m trying to reveal a nuanced situation to you that’s closer to the truth beyond the buzzwords you’ve memorized” will go much farther than trying to engage directly in the culture war with your own buzzwords and tribal language.

AI is unlike most other culture war issues in that a lot of false claims about it can be disproven by just having people sit and play with a model for a bit and see for themselves what it’s like. Inviting more people to do this can be a fun role to play.

There are a lot more guides like this that are still pretty great but out of date. 2 years is a long time in AI! Mine will be out of date soon too.

Contents

• Large Language Models

  • Things they’re good at

  • Things they’re not good at

  • Language

  • Avoiding hallucinations

  • Models I use

  • How I use LLMs

    • Deep research

    • Talking to ChatGPT voice mode while traveling

    • Building a basic narrative of a new field I don’t know much about

    • Aggressively asking follow-up questions

    • BOTECs

    • Summarizing papers

    • Flashcards

    • Solving problems around the house

    • Vibecoding

    • Automating work tasks

    • Life advice

    • Career advice

    • Clearing ugh fields

    • Editing and fact-checking writing

    • Debating and learning about rival viewpoints

  • Some other thoughts on chatbots

• Images

  • ChatGPT/Dall-E

  • MidJourney

• Music

  • Suno

  • Jammable

Large Language Models

I use LLMs hourly at this point. I mostly use them to learn, so a lot of what I say here will continue on from my post on how I learn.

The value you get out of LLMs is limited by their tendency to hallucinate and their sometimes very shallow knowledge of topics, but it’s also often limited by the creativity of your questions and how genuinely you want to learn. There have been a few topics I’ve been thinking “I should really know more about this” for years, and when I actually sit down with an LLM to learn about them I get extremely bored and averse to reading. A hard truth is revealed. I have this basically utopian technology: a smart patient teacher I can ask to clarify anything I want in any style I want, and I still don’t want to learn about the topic. I just want to be seen as a guy who’s learned about the topic. LLMs can be an unpleasant window into our real motivations.

Once I work through my real motivations and start learning, it helps to sit back and let my mind go blank for a moment. There are so many ways to approach a topic with an LLM and I want to be open to new ideas. If I want to learn about the deep learning revolution, I could ask Claude to write me a series of limericks or flashcards or just go all in and write me a textbook1 catered to my exact level of understanding. I could poke it with questions about each individual topic it wrote about that I didn’t perfectly understand, or feed it papers and text from other authors on the topic to clarify what they mean. We’re probably all just scratching the surface of useful prompts.

I think the most useful way to think about LLMs is as if they’re each an extremely massive (metaphorical) database2 of almost all expert knowledge in all fields, with a very smart friend to act as a guide to that database. Google + the internet is in some ways a database of most human written knowledge and thinking as well, but it takes much longer to get the same information out of it because you don’t have the friendly guide that an LLM provides. Your LLM friend is extremely good at drawing out the data from this imagined database and presenting it to you in the way you like. They’re good at doing some basic reasoning with it. Nothing super advanced though. They’re very bad at drawing new knowledge out of the data.

I think a lot about this observation while I’m using LLMs:

One question I had for you while we were talking about the intelligence stuff was, as a scientist yourself, what do you make of the fact that these things have basically the entire corpus of human knowledge memorized and they haven't been able to make a single new connection that has led to a discovery?

Whereas if even a moderately intelligent person had this much stuff memorized, they would notice — Oh, this thing causes this symptom. This other thing also causes this symptom. There's a medical cure right here.

Shouldn't we be expecting that kind of stuff?

- Dwarkesh Patel interviewing Dario Amodei

I can expect LLMs to be amazing summarizers, but not to discover fundamentally new knowledge for me and make new observations. They often lack the subtlety and visible love for the subject of human nonfiction authors. Despite these limitations, having an infinitely patient helper can be surprisingly useful in a lot of areas of learning and life. It just takes a little creativity to figure out where. You can pepper them into the margins of your learning and generate a ton of value, but I don’t expect them to be a main way I learn things.

Things they’re good at

• Fields where there’s clear expert consensus, but that might not have clear routes for outsiders to easily learn that consensus.

• Back of the envelope calculations for problems where there’s publicly available data to pull from.

• Common knowledge stuff in general.

• Summarizing documents and drawing out important points.

• Answering lots of follow-up questions about a topic.

• Some creative tasks, like spoofing song lyrics. I wrote most of my AI musical Rumsfeld! using Claude.

• Coding (compared to me).

• Video mode in general, especially for repair tasks around the house.

Things they’re not good at

• Very specific knowledge questions about very specific not-well-known fields where access to information is very limited.

• Some creative tasks. I don’t like LLM creative writing.

• Vague topics where there’s not much consensus. They usually only speak in generalizations with these.

• I enjoyed Gavin Leech’s post about all the things he doesn’t like about LLMs. He’s much more bearish on them and a lot of what he wrote contradicts what I say here, so I’d recommend his post as a good counter-weight.

Language

Default LLM language can be annoyingly general. You can fix this by going to their personalization option and telling them to only respond very directly.

This is especially important for voice ChatGPT’s voice mode. ChatGPT’s trained to speak more naturally in voice mode, which often means more vague general language. Making it clear that you want direct language can save a lot of time.

In general, if there are aspects of the LLM’s responses you don’t like, consider asking it not to do that in the customization option. You might be surprised at how different you can make it. Getting creative with prompts can also fix style issues. If you don’t like an LLM’s output, it’s usually easy to just say “rewrite that in (style I like).”

Avoiding hallucinations

"You can't trust anything LLMs say" is starting to have the same flavor as people in the 2010s saying you can't trust anything you read on Wikipedia. It’s true that both can make mistakes (chatbots do still make many, many more mistakes than Wikipedia etc.), but saying this is a sign that you don't use them enough to notice how low the error rate is.

If you took someone from 1990 and put them on the 2025 internet, it’s likely that they’d get overwhelmed with fake or strange ideas, at least for the first week. They might have trouble understanding that even though anyone can edit Wikipedia, it’s one of the most trustworthy sources online, whereas a lot of official looking news websites where only specific people are allowed to post actually have a lot of lies.

Chatbots are like this. Getting good at detecting when a chatbot might hallucinate is kind of like getting good at navigating Google and avoiding lies and conspiracy theories. They’re a strange new situation and we haven’t developed a lot of intuitions for how to navigate it. After using chatbots for a bit, you develop intuitions for the places they’re more likely to hallucinate.

You should assume that LLMs are great summarizers and synthesizers of most human knowledge, but if you ask them to draw out too many new ideas from that knowledge they’ll often fail. Writing nonfiction with patience and love for a topic produces some of the best writing that exists, and LLMs noticeably fall way short of that. When things get very abstract, they’re bad at knowing what they don’t know. This will probably change in the future, but for now LLMs are still very limited. Basically avoid asking LLMs to come up with new complex knowledge, or asking about very specific knowledge that isn’t available as an understood expert consensus online, and a lot of issues with hallucinations go away.

One other specific place to watch out is anything that has to do with the letters in specific words. LLMs learn by encoding chunks of words (not individual letters) into a separate mathematical language they can understand. This means that they’re not 100% clear on the letters that make up those chunks of words. A famous problem that some chatbots still struggle with is “How many R’s in Strawberry” although progress in the question is moving fast. This means you should be a little careful if you’re asking for things that involve counting characters and similar specific questions.

You’re probably going to run into hallucinations. Try to think about where and how this can be useful to you even if it occasionally fails. These models are far from perfect. If they were perfect things would get weird. If it’s really really important that you know that something is 100% factual, you can ask the model for sources for everything it says and double check them.

Models I use

These are what I use right now. I expect they’ll change a lot over the year:

• ChatGPT - Deep research, reasoning problems, voice & video mode, internet search, images.

• Claude - Writing and reading long text, summarizing papers. I love Claude’s aesthetics, and because it has my favorite writing it’s my favorite model to use.

• Gemini Pro - Coding, honestly should be doing a lot more with this. It seems like the best overall model right now, except for deep research.

• Perplexity - Internet search.

• NotebookLM - This is on my to-try list, will get to it soon. Seems really useful. I think their podcast tool is fun but basically a gimmick right now.

How I use LLMs

Deep research

I think deep research is the single most useful thing chatbots do right now. ChatGPT’s is the only one that’s truly great (I was really unimpressed by Gemini’s deep research recently, it’ll get better though). If there’s no clear overview of a complex topic, I can get a better one than what I can find online by asking Deep Research for it. It’s great at doing a lot of back of the envelope calculations to come up with estimates for important statistics about the real world. Here’s a report I asked it to do on the overall situation with all factory farmed chickens in the US and their overall welfare. It’s the best single thing I’ve read on the topic.

I’ve asked it to generate reports for me on complex topics I’m trying to communicate better about. It was really useful in writing my Using ChatGPT is not bad for the environment post. I double-checked all the numbers after, but it saved me hours of scouring the internet. If I could convince everyone to try only one chatbot tool or trick, ChatGPT’s deep research would be it.

Talking to ChatGPT voice mode while traveling

If you’re walking somewhere and it won’t disturb the people around you, you can talk to ChatGPT voice mode as if you’re talking to someone over the phone about a topic you’re interested in. I do this on my walk to work. This is maybe the most engaging way I know of learning new information over audio. It’s much more engaging than an audiobook or podcast because I can trick my brain into thinking it’s in an active conversation with someone and I feel more obligated to listen closely to what it’s saying and follow up with direct specific questions. This pressures me to pay a lot more attention to what’s happening in my brain, and notice what I don’t actually know much about, and follow that thread with more questions. It’s great for retention.

Building a basic narrative of a new field I don’t know much about

When I’m trying to learn about a new field, it’s helpful to have a kind of skeleton I can attach facts to later; a single coherent narrative of what’s happening. Even if that narrative is mostly wrong or oversimplified, it’s helpful to start there and make it more complex later. When I was trying to learn a lot more about China, I started with the book Wealth and Power, which is a series of profiles about the most important individuals in 19th and 20th century Chinese history. I don’t think understanding history as a series of profiles of people is super useful for actually understanding the world, but having a pile of simple narratives to build a broader understanding of an era is a good first step, otherwise I find that it’s hard to retain new more complex facts.

A prompt that works well here is “Tell me the basic story of (topic) as if you’re explaining it to someone completely new to it, and so that by the end they understand most of the key ideas.”

One of my favorite prompts to use has been to ask for a series of profiles of important people in a new field and how they contributed, and to frame them as part of a single overall narrative of how the field developed. This can help use my need for a simple narrative to get started in understanding the field as a whole.

When I’m ready to go deeper I sometimes prompt it with “If you were to write a textbook introducing (subject) to someone with (detailed description of my exact level of understanding and background), what would each chapter be titled? Include a description of the contents” and then ask it to expand on specific chapters I’d like to read.

Aggressively asking follow-up questions

We’ve all been socialized against asking a lot of specific follow-up questions when presented with new information. It feels annoying to do, but it’s really important. It’s very easy for me to read pages of books before realizing that I don’t actually have a deep understanding of what I’m reading. I can nod along to terms I feel like I understand, but if I actually try to test myself on what they mean I realize I’ve just been playing pretend.

Asking follow-up questions that make me feel a little stupid is the way around this, and LLMs are always available to answer. If I even vaguely don’t understand a chunk of text, I’ll just copy it into a chatbot and ask it to explain it as if I’m new to the topic. This almost always helps a lot. It makes reading take longer, but I leave retaining and understanding a lot more.

It’s possible now to upload whole articles to an LLM and then ask questions about the article as you go as if you were interviewing the author. This can be especially useful for very technical topics you don’t have a strong background in. Econ papers in general have always been a challenge for me. LLMs make them much easier. Chatbots are also surprisingly good at continental philosophy. I recently had a great conversation about some essays by Heidegger where the LLM consistently helped me understand them better.

Follow-up questions in conversations with LLMs are always useful too. In general, remembering that the goal of reading is actual understanding rather than just making it to the end of a page matters a lot. Getting into the habit of grilling AI on specific topics until you’re sure you completely understand them can help you avoid just pretending to learn.

LLMs can be useful in building up your tolerance for asking questions that feel stupid and low status but that will actually help you learn. A key skill for adult life.

BOTECs

Chatbots are great at back of the envelope calculations. If you’re trying to get a rough estimate of a large number of something in the world, chatbots usually produce great results. If the LLM has a reasoning model (ChatGPT has o1 and o3) make sure to use that for the botec.

Summarizing papers

Before I read a paper I’ll often upload it to an LLM to give me an 80/20 of the contents. Knowing the overall message of the paper makes it much easier to retain when I’m actually reading it.

Flashcards

Spaced repetition is one of the most effective ways of actually learning and retaining new information. You can have the LLM generate a large collection of flashcards on a specific topic you’re trying to learn more about and ask it to format them to easily add to Anki, or Obsidian’s flashcard plugin.

Solving problems around the house

I solved a simple plumbing problem by pointing the ChatGPT camera mode at it and explaining what was happening. It gave me a clear set of instructions for how to solve it. Thanks ChatGPT. It can pick up on a surprising amount of context over video.

Vibecoding

I’m planning to sit down with Gemini soon and vibecode a browser-based game. I have some coding experience but this will make it much easier. As Andrej Karpathy said:

If you don’t know what to do with the code the model generates, just ask the model!

Gemini Pro 2.5 seems to be the clear favorite for coding right now.

This is a great article on what you can do with vibecoding.

Automating work tasks

You can connect Claude and other LLMs to databases and other files for your work using Zapier. This can take table data, feed it to Claude with specific requests for what to do with it, and then fill in Claude’s answers in a separate column in the data. I’m only just starting to scratch the surface for what we can do with this.

Life advice

A lot of good life advice usually just requires someone who doesn’t have your specific neuroses examining your situation from the outside and saying something pretty obvious. It usually doesn’t take a profound personal connection or insight into every last detail about you. If you’re struggling with something in your personal life, go to a friend or therapist, but also consider pinging a chatbot. It just takes a few seconds.

A lot of people my age underestimate how often young people use TikTok, Instagram, and YouTube for life advice. In my opinion this is often pretty bad. A lot of that content is insight porn, something to simulate the experience of having a deep insight without actually giving you useful knowledge. It’s also optimized for maximum attention, so more dramatic ideas win out over level headed advice. It’s very noticeable how a lot of the most popular podcasts among young men now frame almost every problem as “They’re lying to you about” (topic).

This works well for the algorithm, but seems bad for keeping people ideologically sane.

I’d be willing to bet that if someone searched TikTok, Instagram, or Youtube for life advice on school, friends, dating, exercise, or difficult emotions, there’s a very high chance that the results they’d get would be much worse than what an LLM would give them, including ideas that would be actively harmful. None of this is a replacement for a real person, but of the other options available LLMs clearly win out. They have the advantage of not being designed to hack our dopamine.

If there’s a problem you’ve been having trouble talking about with other people, try typing a lot of details about it into a chatbot. You might be surprised.

Career advice

Career advice, like life advice, often doesn’t require a crazy amount of insight into the human subtleties of your situation. A lot of good career advice is good for a lot of people, and people just have trouble accessing it because they can’t tailor it just slightly to their specific circumstances at work. I’d trust Claude’s career advice for someone over a lot of online articles. If you’re on the market, try writing a lot about the specifics of your career situation and goals into a chatbot and see what happens.

Clearing ugh fields

An ugh field is a task that you don’t want to do, and that’s causing you so much shame for not having done it that even thinking about it is painful. From Rob Wiblin’s blog:

From this, your brain gradually learns that thinking about this task is the mental version of stubbing your toe. Just as your brain learns to avoid whacking your foot into things, it learns to find creative ways to prevent the task you’re avoiding rising into your conscious awareness.

I’ve had a few ugh field tasks recently I’ve been really averse to thinking about. Just planning out how to deal with them is tough. I realized that I could just ask an LLM “Hey I’m trying to do (task) and really don’t want to. Here are all the details. Please break down exactly what I need to do for each step and make a checklist for me, including every minute little step. If anything requires external information, find the website where I can get that information” and then just unthinkingly follow the checklist. Going into this saying “I’m basically going to be a big baby about this and get the most help from the AI I can, even though this is all really silly” can actually give you an easy way through an otherwise painful experience and make the whole process much faster. Clearing ugh field tasks can be one of the most liberating experiences in everyday life. A little helper tool to make those easier is so nice.

Editing and fact-checking writing

I actually don’t especially like any AI’s writing style. It’s good for learning, but not good for convincing essays (and definitely not for fiction). However, if I’m writing something, LLMs can be a useful way to review and edit. Uploading your final doc to an LLM and asking for a full list of all grammar mistakes and ways to improve the quality of the writing can be great. I disagree with over 50% of LLM advice on how to improve the quality, but it still gives me a lot of useful ideas and updates.

You can do the same to fact check something you’ve written (or any other writing). Just upload the doc and ask the LLM to fact-check every significant claim. It’ll produce a useful list of its best guesses at where the piece is wrong.

Debating and learning about rival viewpoints

If you feel like you don’t fully understand where the other side of a debate is coming from, you can ask a chatbot to behave as if it believes the other side and argue against your points as convincingly as possible. You can go back and forth with it for a while. This can draw out a lot of useful ideas about the other side you might not have noticed by just reading about it on your own. It’s sometimes challenging to find how the other side would respond to your specific objections if you don’t know anyone who actually believes it.

Some other thoughts on chatbots

Chatbots are anti-addictive

An underrated aspect of LLM apps is that they aren’t built to hack your dopamine, and often require thoughtful focused reading. When I use ChatGPT or Claude or Gemini I sit down knowing I’m going to read a lot of sometimes dense material that will require calm focus. Looking at the other most popular apps, I’m happy that ChatGPT is where it is.

AI can be amazing without being magic

Some people interpret me saying “AI is so useful! It’s great! I use it all the time!” to mean “AI can answer every question and is always useful! Everyone has to like it and use it.” which isn’t the case at all. AI is a really useful tool in the same way previous tech has useful. It can be a crazy step up while still having clear obvious limits. YouTube is also a crazy useful tool for learning. If I found out someone had never even considered using YouTube to learn something, I’d hype it up and push them to try it, because they’d obviously be missing out. But YouTube also has a lot of bad/fake info and other noticeable problems. If someone responded “But look! There are all these fake bad videos on YouTube! And it’s using as much energy as whole countries. It has a ton of stolen content on it too. You should really stick to reading books.” I’d react by saying that they were overreacting to a fairly harmless and clearly useful new tech that has a lot of very obvious benefits if you just spend the time to learn how to avoid the downsides. A lot of AI discourse feels like living through the early days of YouTube again, only this time everyone is much more brittle and conspiratorial about new tech and are constantly assuming it’s trying to trick them.

Images

One of the clearest signs of AI progress is how far AI image generators have come in the last few years.

Prompt: A cat dressed as a computer programmer

The progress in AI image generation was so rapid that you quickly saw common knowledge about it develop, and then everyone acted like that common knowledge was going to be around forever, and then in a year it was invalidated. “AI can’t draw hands” is still a popular idea but isn’t the case anymore.

ChatGPT/Dall-E

ChatGPT’s advantage in image generation is its understanding of very subtle requests, picking up on the exact look of a picture, and transferring relevant parts of that picture over into the new style. Here’s a picture where it turned Neutral Milk Hotel into muppets:

Take a second to look at all the little details the AI picked up on and made extremely subtle decisions about what muppets would look like with the exact same details.

Here’s a Mao-era propaganda poster of me over a crowd of farmed animals holding copies of Reasons and Persons:

My favorite use of AI images in general is making extremely personalized images like the Maoist poster above for friends. You can include a ton of very specific details about them in the picture. I’m not sharing these for their privacy, but you can make images of Maoist posters or action figures or Renaissance art that feature your friends doing the very specific things they like.

ChatGPT works great even when you’re just prompting it without any reference images. Here’s “Solarpunk Dunkin’ Donuts” in different styles:

It’s also pretty good at making comic strips:

ChatGPT is good for very specific graphic design that needs a lot of context, or fun pleasant images of specific people and places in different art styles to make them happy. Both are great but limited.

I personally prefer MidJourney’s aesthetic style to ChatGPT. ChatGPT follows instructions well, but often feels kind of rigid. Here are two examples with ChatGPT on the left and MidJourney on the right. In both cases I prefer the look of MidJourney.

Prompt: An AI large language model in the sky. Below it people exist in a confused system where they're constantly changing their minds. Illustration style

Prompt: A landscape architect sketch of a concrete brutalist monument in a forest

ChatGPT is better at capturing all the subtle details I might be asking for, but it often just doesn’t look as great as MidJourney, even if MidJourney deviates from what I’m asking for. If I’m just trying to make an image I like a lot (that’s not an image of a person or place I know) I’ll usually stick to MidJourney.

Prompting

Finding styles that work and that you like

ChatGPT is better at some things than others. It can turn people into muppets, but has trouble putting their faces on Mount Rushmore etc. I think its impressionist paintings mostly look tacky and often obviously AI generated, but really enjoy looking at the cartoons it can generate. There’s a lot more to explore. It’s pretty new as of writing so I’m sure people will find more good prompts.

Refining an image

If you don’t like how an image looks or want a specific detail changed, ask ChatGPT to modify it a few times. Just use very clear specific language. I sometimes combine the final results from a few different images by editing them together separately or just uploading them to ChatGPT and asking to combine the vibes of each. Here’s ChatGPT’s result when I asked it to combine the vibes of images of Biden and Trump:

The written detail it can handle is limited

You can include a surprising amount of very specific details for what you want your picture to look like, but it’s still limited. Experiment with finding the limit for what you’re trying to do. You can actually smuggle in a lot more information if you draw a basic version of the picture you want ChatGPT to create, upload it, and then say to add a lot more detail and describe the specific style of art you’d like. It can handle a lot more detail and context sent over images than text.

MidJourney

MidJourney’s my favorite AI image generator. It’s the most reliable at actually making images I like looking at.

Its controls are specifically designed to modify images, so you get a lot more options for the layout of your image, and you can edit it in much more specific ways than ChatGPT. The basic plan costs $10/month, it’s worth paying once, experimenting with it to see if you like it and deciding if you want to keep using it.

The latest version of MidJourney is the first image generator I’ve seen that can make extremely convincing looking images of people and places. I wouldn’t have been able to tell that any of these were AI unless I looked really close:

Image generators can easily create hands now. The common wisdom “Just look at the hands, AI always messes them up” is over:

Here are a few images I’ve made recently that I liked a lot. I mostly use these for desktop wallpapers, covers for other stuff I’m making (like blog posts), or just keep them to have and look at:

Here’s an album with some more pictures I’ve made that I liked. I share a few more in my post about AI art.

Prompting

I haven’t found any special tricks to prompting MidJourney beyond getting good at the controls (especially the edit feature), giving it a sense of what style of art you like, and exploring a lot of different art styles to see what it’s especially good at.

Music

Suno

From what I can tell Suno’s the best AI music app right now.

Prompting

Suno lets you write lyrics and add tags for the style of music you’d like. you can also opt to make your song entirely instrumental.

Lyrics

It’s very important in the Suno lyrics section to specify which parts of the song are a “(verse)” or '“(bridge)” or “(chorus)” at the beginning. This doesn’t always work but a fun trick is to also add “(spoken word)” for the intro if you’d like your singer to just talk at the beginning of the song. I did that here.

If you include parentheses around words in the main lyrics line, the model will read that as backup singers, which can add fun texture to songs.

Styles

Suno has a spot below the lyrics where you can write a 200 character description of the style of music you’re looking for. It’s built to read this as a series of individual tags rather than whole sentences, so you should write things like “Glitch, triphop, nostalgic, eerie” instead of “A glitch song with triphop elements” etc.

If I want to make a song sound like an artist I like, I can just go to ChatGPT or Claude and ask “Please write me a 200 character Suno prompt to make the song have the same sound as (band). Do not use any proper nouns, make it very info-dense, as a series of tags instead of whole sentences. Separate everything by commas.” I generated the first AI music I really enjoyed listening to by doing this with Boards of Canada. ChatGPT’s description was:

Lo-fi analog synths, VHS warble, warm tape hiss, dreamy nostalgia, slow breakbeats, deep reverb, eerie speech samples, minor-key pads, hypnotic loops, surreal, hazy, downtempo, 90s IDM.

Another good place to find style ideas is to click “Home” and listen to the popular songs other people have made, and if you hear any styles you like look at the tags the person used to make it.

Original music

Writing lyrics

The only song I’ve made on Suno that I actually sat down and wrote all the lyrics for is Get Bigger Than the Carnists. For most other songs I asked Claude to write lyrics (and modified them a lot, you need to push Claude to not use overly general language, even in music!). With Claude’s help (and ChatGPT for prompt ideas to make Suno use the right style of music for each song) I wrote and published Rumsfeld! The Musical in a few hours of playing with the model. I found myself actually listening regularly to this punk cover of Thoughts for the 2001 Quadrennial Defense Review, and Defying Intelligence.

Covers

Suno really cracked down on copyrighted material, so you unfortunately can’t just take lyrics from songs you like and make them different styles. Before they did that I made a few. I found that country music is probably the best-sounding genre for Suno covers, so most of these are some variant of country:

• Hot to Go! - bluegrass

• Golden Boy (Mountain Goats) - hillbilly, and in French (you notice how good these models are when you hear a song in another language)

• Old timey Espresso

• 365 - country

• Northfield (a favorite sacred harp hymn) in a bunch of different styles

Jammable

I’m closing this with the most tasteless way I use AI: making covers of songs I love with stupid character voices. Jammable makes it really easy to get stupid AI covers of songs you like with a lot of characters.

Here’s Patrick Star singing Trouble by Cat Stevens:

Maybe the most tasteless thing I’ve created is Plankton singing Gardenhead/Leave me Alone by Neutral Milk Hotel:

There’s something stupidly cathartic about it. The song was one of my favorites as a teenager and I listened to the Plankton cover over and over one day. Inexcusable.

I don’t use AI art much, but when I do it’s nice to easily create images I like looking at, even if I know they might not have a deeper meaning or intention behind them. I’m not trying to convert you to liking AI art, I just don’t think you should worry about it as a serious problem, and I want to share what I think is interesting about it.

The quality of AI art

Pleasant pictures vs. serious art

I think of there being 2 categories of art:

1. Art that's challenging or interesting or emotionally engaging

2. Art that's just pleasant to look at without any real depth

This post will only be about the second category. I haven’t seen any AI art that I’d consider challenging or interesting, but I’ve seen a lot that I find pleasant. I wouldn’t want to automate away art that’s intentionally created by a human to challenge me or draw me into ways of thinking about the world I couldn’t otherwise access, but a lot of my (and others’) engagement with art isn’t like that. A lot of everyday people are often just looking for pleasant images or sounds. It’s okay that people spend time listening to lofi hip hop beats when they want to relax, and it’s okay that it’s easy and cheap to mass produce those. I don’t see why we wouldn’t want to produce a lot of additional pleasant pictures on demand if it makes people happy.

It might be that human artists can use AI to produce challenging work that would fit the first category. In that case I’d consider the AI to be just another medium, like paint or clay, that the human is using and refining to get at a very specific aesthetic experience. I haven’t really seen that type of art yet but would be open to it if and when it arrives.

A lot of AI art is bad because the everyday people making it have mediocre taste

I think a lot of what gets shared as impressive examples of AI art is really ugly. It’s what critics call “AI slop.” Here are some examples:

Looking at what’s popular on MidJourney right now:

All these images are sickly sweet, weirdly shiny, and “cringe.” A lot of people who haven’t looked into AI art much assume that all of it has this “attempting to be epic” look, like something an 18 year old on Reddit would be excited about rather than a more serious adult with a more subtle appreciation for complex and challenging art. They’ll often say the sickly sweet AI art lacks human depth, and assume the reason it looks that way is that it was created by a machine rather than a human. I agree that the images above, and a lot of what’s shared as “impressive” AI art, looks like this. It’s bad, but I don’t think the reason it’s bad is because it was created by AI.

I think what’s actually happening here is that a lot of people sharing AI art are just average people with an average person’s level of appreciation for art, which is often pretty cringe and far from what artists and art critics would consider high quality. Obviously this is elitist, but I think it’s true.

If you look at communities of everyday people sharing art they like, it’s often pretty tacky and embarrassing. Where can we find some examples of everyday people sharing a lot of art they like? It’s important to not look at a community of artists or art critics, or people spending a lot of time trying to refine their own taste and display it, otherwise it’s not representative of what average people are looking for. We could look at a large forum of everyday people sharing desktop wallpapers they like. This is one of the most common ways everyday people display art. Artists and art critics won’t be the predominant users, it would be mostly everyday people sharing images that make them happy. What would this art look like? We can go to Reddit.com/r/wallpaper to find out. With almost 2 million users it’s the largest forum I know of for sharing images that everyday people enjoy looking at, with no competition to display their refined taste. Here are the top 5 most upvoted images on /r/wallpaper of all time:

I don’t like any of these images at all, and I’d go so far as to say that I’d be embarrassed to have them as my desktop background. It would signal that I’m not “with it” and that I’m lacking a depth of appreciation for art that mature adults have. Again, a very elitist thing to say, but I guess I’m just kind of a snob about the art I consume and surround myself with.

The same thing happens if we look at /r/art, with 22 million members. This one is more directly focused on art, but in my opinion the top images of all time also basically look like slop:

A lot of the AI art that’s getting made and shared is by people with an average level of art appreciation, so we should expect most of it to not fit our high standards.

When critics of AI art say that it feels inhuman and artificial, it’s hard not to assume that they would have the same reaction to the /r/art or /r/wallpaper pictures. I think what’s actually happening here is that critics of AI art are just reacting to the tackiness of everyday people’s taste, and incorrectly blaming AI for that tackiness. AI has in some ways democratized access to creating art, so we should assume that a lot of its output will be much lower quality by the standards of the elite.

The tackiness of common taste isn’t new. What would you guess is the most popular art print of the 20th Century, the one that the most Americans had hanging in their homes and offices? Maybe it was something that everyone knows, like a da Vinci painting, or something more bold and modern like Warhol’s Campbell’s Soup Cans, or something simple and calming like The Starry Night.

All of these are wrong.

The most popular art print of the 20th century, the one that the most Americans hung in their homes, was the 1922 painting Daybreak by Maxfield Parrish:

There was 1 print of this made for every 4 American homes in the 20th Century.

Something funny about Daybreak is that it has the feel of AI slop. It looks like it’s trying to be epic and over-the-top. The color scheme feels more like it’s supposed to just make my eyes happy rather than invoking a more subtle mood. The two figures even have a shine to them. This makes me suspect that what’s happening with AI slop is that the masses’ taste is, for the first time, being much more widely manufactured and shared, and the results aren’t pretty to me and other art snobs. The demand for slop was always high, it’s just finally being met with adequate supply. If you find yourself assuming that all AI art looks bad and cringe like this, consider that this might mostly be coming from a kind of elitist reaction to the tastes of everyday people and what they’re choosing to do with AI, not the full capabilities of the AI models themselves.

It might be that AI does have some inner tendency toward producing slop because it was trained on so many images. It’s very likely that a lot of the images it was trained on were just bad art!

If you play with an AI image model enough, it becomes easy to generate art that you like

These are some images I’ve generated on MidJourney that I personally enjoy looking at. I’ve posted more here. You might not like them! I’m using these as examples of what an AI image generator can make that doesn’t have the “inhuman slop” look to it, not examples of objectively great art.

Impressionist painting of Western Massachusetts

Minimalist painting of a forest

Photorealistic graffiti of a sphere

Watercolor farm

Watercolor cassettes

West Virginia painted in the style of Lawren Harris

Modern art painting of a city

Landscape architect sketch of a brutalist monument in a forest

If you don’t like these, try playing around with MidJourney for a while and see if you can make the type of images you like. If it doesn’t happen to work at first, experiment for a bit before giving up. Try getting used to the MidJourney controls and editor options.

My goal here isn’t to convert you to liking AI art, it’s just to show that the “slop” vibe of so much AI art isn’t built into the models themselves and they’re capable of a lot more.

I think we should be skeptical of simple humanist narratives that there’s something fundamentally magic happening in the human experience of art that AI can never access. Modern AI is built to pick up on extremely subtle vibes, patterns, and associations; not to impose clearly written explicit rules and algorithms. It shouldn’t be surprising that it can pick up on much more subtle decisions human artists make and recreate them without being able to clearly articulate what it’s doing. I worry that people saying that AI art is all low quality because it’s lacking some mystical human element might just be imagining that humans have some deep magic that machines can’t eventually access. Because I don’t believe in magic, I don’t believe that.

I don’t think AI models can produce art at the level of the best artists right now, or be as creative and groundbreaking as human artists, but I do think they can produce images at a higher aesthetic level than what most humans (including me) can create. This seems useful and exciting and opens up a new world of fun ways to play with and generate aesthetic experiences.

Death of the author

When I went to college, if you didn’t believe in “the death of the author” you were considered kind of dumb. The idea is that critics should not defer to an artist or author’s intentions for their art as the final arbiter of what the art itself means. Art was understood as having a life of its own independent of the person who happened to create it. This implies to me that we should be more willing to engage with images on their own merits rather than looking for the deep intentions of the human who created them. If you accept the death of the author thesis, it seems silly to declare that only humans can produce “real art.” If we assume that the artist’s intentions don’t matter and we should allow ourselves to be affected by the piece as it exists on its own, we should be more open to work produced by machines without human intentions at all.

Sometimes machine-like inhuman art can be great!

It’s hard not to think that a lot of people criticizing art because it was made by a machine would have a similar negative reaction to some of the best artists of the 20th Century. Andy Warhol specifically presented his art as a challenge to the idea of individual artistic genius, treating art as a process that could be mechanized and even depersonalized. His Factory churned out silkscreens in a way that intentionally blurred the line between mass production and fine art, and he openly embraced the idea that his work was about surface, repetition, and commercialism rather than deep personal expression. If Warhol could take mass-produced imagery, apply a process that involved minimal handcrafting, and call it art, why should AI art be any different?

Is AI art stolen?

AI doesn’t memorize the images it’s trained on and plagiarize them later

AI is trained on the work of millions of artists. It’s fed an incredible number of images from across the web to build their intuitions about what we want it to produce, and then produces art based on the vibes it picked up from those images. Is this a form of theft against the human artists?

What normally counts as plagiarism

I go to an exhibit and see some paintings of trees that look like this:

I take inspiration from them and paint this image of a tree:

You’ll notice this tree has a bit of the vibe of each of the ones I saw at the exhibit. The branching pattern of the first, the basic color scheme of the second, the more thick trunk and surrounding trees of the third. Have I plagiarized the other artists’ work?

Obviously not. It’s very normal for artists to be inspired by other art they saw and incorporate the basic vibes they like into their own work.

Now instead I produce this painting:

This is very obviously plagiarizing the second painting from the exhibit. There are some extremely slight details that are different (falling leaves) but it’s so obviously a copy of what I saw that it’s a form of plagiarism.

What is the difference between these two? Basically what makes something plagiarism is whether you cross a line in how much you’re taking from a very specific piece of art, rather than a lot of different art. You can incorporate the vibes of a lot of previous pieces into a new piece, and that’s considered creative. However, if you only slightly modify a single piece and are clearly only drawing inspiration from that one piece, it’s plagiarism. I’ll stick to this definition of plagiarism for the rest of the post.

You may have correctly guessed that I generated all these images with AI. With the first example, I asked it to incorporate the general vibe of the original 3 trees into a new tree. It did a great job of incorporating the overall vibes without making an explicit copy. For the second, I asked it to make an almost exact copy of the second tree with some very slight changes. It did that well too.

My claim here will be that if AI works like the first example, where it picks up on the subtle interesting artistic vibes of a lot of different works and incorporates them into new works, it’s not stealing those works, in the same way I wouldn’t be stealing from art I was inspired by. It would be doing exactly what human artists do all the time to create new art. However, if it’s just storing a ton of images to individually copy when creating something new, it is stealing that art.

How AI models work: they’re entirely inspired by vibes

It’s really really worth taking some time to get a basic sense of how current AI models work. I have a separate post on how to do that. If you don’t want to read all that, I think the best single thing you can do is watch this 8 minute video:

A key takeaway from the video is that deep learning models do not save specific information in a specific place and then give it back to you later as output. I've found that a lot of people assume AI language models have every possible answer to every possible question memorized and stored in different files, and all the AI is doing is calling up those saved answers. This is wrong. People do the same for AI image generators. They assume the AI has every image it's trained on saved, and when it's asked for a specific image, it finds all the images it's trained on, modifies one a bit to avoid copyright issues, and then gives that one image to the user. This is also wrong. If this were true, the AI would be plagiarizing art in the same way my painter did in the last example: taking a single saved image someone else produced, modifying it slightly, and presenting it as the AI's own work. What's actually happening when AI trains is that it’s picking up on all the more subtle patterns and statistical relationships in millions of images or pieces of text, and then constructing new material based on these subtle patterns it’s picked up on, like the painter example where the painter is inspired by, but isn't directly copying any one of, the tree paintings.

When AI is trained on art, it doesn’t have anything that allows it to just “save” an image in the way we could save a file on our computer. Instead, it's given a bunch of examples of different types of art, and then it adjusts an incredibly large number of simple connections between "neurons" (simple computational units that process information by receiving inputs and producing outputs). These connections are called "weights." The latest image models have billions of these weights and similar values, so they can pick up on billions of individually subtle patterns and relationships in the data.

The motivation for using this type of model over specific instructions humans write is that we can allow it to train itself and pick up on subtle rules that we ourselves can't articulate through words or code. By continuously adjusting its weights and nodes as it trains on millions of examples until it can accurately predict what specific types of art are supposed to look like, it can learn an extremely large collection of subtle patterns, heuristics, and vague rules that we can also pick up on but can't articulate.

If you believe that AI training on artists’ work is equivalent to theft, you should also believe that it’s theft for a human to observe these paintings:

and then paint this:

The exact same thing is happening in both the human and AI case.

Why are we all suddenly defending intellectual property?

I was always at least a little sympathetic to more radical ideas about how there maybe shouldn't be any intellectual property at all. Property rights make sense when there are clear reasons why you would be harmed if someone else used something you own. I’m very happy that I have property rights to my toothbrush. I don’t want other people using it. Intellectual property is by definition not really a scarce resource once it’s created. If I write a blog post, I’m not affected by people copying and pasting it somewhere else. Maybe they would get some credit that I’d deserve instead, but that’s very different from me being deprived of something physical. I and others can still access the information whenever we want.

Internet piracy works in a similar way. If I walked into a store and stole a DVD, I’m depriving the store owner of a physical object they had to put a lot of effort into acquiring, and that now other people can’t use. If I take it, other people can’t use it. If I were to pirate the movie online, this is more like walking into the store, hovering my hand over the DVD, and creating an identical copy of the DVD out of thin air for myself, and leaving with the original DVD in its same place. I don’t deprive others of access to the DVD. The big issue with doing this is that I deprive the store owner and movie company of the reward for making and distributing the DVD in the first place.

I’ve known a lot of people who are very pro internet piracy but for some reason very anti AI models “stealing” art. This is confusing to me. Either you believe in intellectual property rights or you don’t. The best explanation is that these people might think that by pirating movies they’re attacking large corporations, whereas stealing art harms individual creators, but this seems like a pretty blurry arbitrary line. Many individual artists work for movie companies and pour a lot of hard artistic work into what they do.

While I’m against internet piracy, I think there are good arguments for significantly relaxing intellectual property protections. The main argument for having some IP rights (incentivizing creation of valuable intellectual work) is convincing, but I worry there are too many cases where IP rights are clearly disincentivizing new creative work. However, even if I didn’t want to reduce IP protections, the call for art be “protected” from training AI is not only a demand to keep IP protections as they exist, it’s a call to radically expand IP protections. It’s equivalent to requiring everyone viewing art on a website to sign a legal document saying that not only that they won’t directly copy the art they’re looking at, they won’t even produce anything inspired by the general vibe of what they’re seeing. This would radically restrict the art that gets made. I would personally like a lot more (and more varied) art to exist, and would like our IP laws to incentivize that, regardless of who (or what) is producing the art in the first place.

AI art’s energy use

MIT Technology Review found that AI image generators use 1.22 Wh per prompt.

For a comparison, instead of generating an image with MidJourney, using the same energy you could:

• Run a space heater for 1.7 seconds

• Print half a page of a physical book

• Run a page on a laptop for 2 minutes. If it takes you 10 minutes to read this blog post, for the same energy you could have generated 5 AI art images. If you don’t feel bad about using energy to read this, you shouldn’t feel bad about using energy to generate AI art.

The cost of 1.22 Wh of energy where I live in Washington DC is $0.0002. If the datacenter generating AI art were in DC, you would need to generate 3300 AI art images to increase the datacenter’s energy bill by $1. The average DC household uses enough energy each month to generate 520,000 AI images. This is an incredibly small amount of energy and worrying about it will distract the climate movement from more serious problems.

How should we think about human artist job loss to AI?

Let’s imagine that we invent a way to pause time. It becomes extremely easy for anyone to spend as much time as they like pursuing hobbies they care about, and then unpause time to show off their new skills. Almost everyone spends a lot of their paused time becoming good artists. They get really good at expressing their exact taste using different mediums, to the point that it becomes easy for them.

As a result, the world has a lot more complex and varied art from almost everyone, from almost all perspectives and tastes. The supply of art explodes. This means that it becomes hard to make a living as an artist. Many artists who spent years refining their skills before the time pause was invented are now no better at art than everyday people. They still have their unique perspective and can share their art with others, but because it’s become so easy to access art from anyone and everyone, basically no one is willing to pay a lot of money for it.

In this world, there’s more art than ever, it’s not commoditized, everyone is able to express a much fuller sense of themselves and their individuality through art and learn a lot more about others via art that others make, and it is also mostly impossible to have a career as an artist. Former professional artists get other jobs and only do the art they care about on the side.

Let’s imagine now that you’re running society and have the choice to invent this technology to give people the ability to pause time and pursue what they like. Do you choose to do that?

I see the situation with artists and AI art as being more or less identical to this situation. AI art gives everyday people the ability to produce a lot more art that’s refined to their specific individual taste. It’s like we’ve magically given everyone a much greater artistic ability. It’s obviously different in the sense that they don’t have the same experience of putting the art together over time, so maybe we can think of it more like giving everyone a camera for the first time instead of having to paint and draw scenes that they like. There is no way we can give everyone much more artistic ability without also making it harder to make a living as an artist, in the same way making everyone a fantastic chef would reduce the demand for restaurants.

If you think this is silly because AI art is just punching words into a prompt and letting the AI do the art for you, I don’t think you understand how current AI art tools actually work. They give everyday people a ton of editing power and ability to refine images and music and video over time to fit what they’re looking for. They make a lot of people happy and excited about the art they produce. This is a big deal! It’s like we’ve discovered a new magical artistic medium that a lot of people really enjoy using. Our baseline reaction to that should be positive.

I love art and the people who make it. I’m not out to undercut their livelihood. The problem is that progress in making a product more accessible by definition undercuts the livelihoods of the people who can only make a living if the product they’re selling is scarce. To solve this problem I’d be excited to either redistribute wealth to give people more time to pursue art, or help them find new fulfilling jobs, but I’m not going to ask to pause a technology that’s making the art itself much more available for everyday people to create.

People will often talk about problems with new technology without being clear that the problem is that the technology is making everyday people too free and able to do what they want. Everyday people being given inexpensive powerful tools to do what they want is extremely good, and it also creates problems. The reason why deceptively edited AI videos have become a widespread problem isn’t that there’s something evil lurking in the AI video editors themselves, it’s that AI video editor programs are just very effective and easy to use, and have made it easy for everyday people to basically become special effects artists and video editors quickly. A lot of new societal problems happen when everyday people become more free and empowered to pursue what they want, because some of those people will have bad intentions, but except for extreme cases we should always be pro empowerment and freedom for everyday people and deal with the risks that come with that after. We should think about AI art in the same way. Most of the bad effects of AI art on artists are coming from the fact that AI makes it much much easier for everyday people to create art they like. This is a downside of giving people more freedom. We should learn to live with the downsides of freedom.

In general, there will usually be way more positive spillover effects after automation makes it much easier to produce a lot more of a scarce resource than the negative effects on specific people who might lose work. We should acknowledge the benefits of a new medium that lets everyday people produce an incredible amount of very specific and personal art as if they’ve had years of practice. It could usher in a world where everyone has much easier access to art, at the expense of the people who are (very justifiably!) benefited by the scarcity of artistic talent.

AI art and creativity

It’s a philosophical question whether people are “creative” with AI art, or “make AI” AI art in the first place these, or whether it counts as art at all. When I generate AI art, I think of it as if I “found” images in MidJourney’s vector space. I’m excited that I found them in the same way I’d be excited about finding a neat rock. I don’t think much of my own creativity went into it, and I know it’s different from creating art myself. Curating AI art feels kind of like curating a Pinterest board. I think that can be a neat way to spend time to develop and show off your own taste even if you’re not putting any creative effort into the images themselves.

Conclusion

To summarize my points:

• AI art models mostly produce slop because it’s what everyday people want.

• It’s easy to make AI produce better art, you just need to prompt it yourself instead of relying on the taste of the average user.

• AI doesn’t “steal” art by any reasonable definition.

• Intellectual property rights would need to be expanded so much to stop AI from training on art that they would be draconian.

• AI art isn’t bad for the environment and you would need to generate hundreds of thousands of images every month to have the same effect on the environment as a normal American household.

• We shouldn’t entertain arguments against AI art that would also work against cheap, abundant, good art that anyone could participate in. Human artists are able to have careers because their talents are currently scarce. If the bad thing about AI art’s effect on artists is that it causes artistic talent to cease to be scarce, the good outweighs the bad.

One final point that didn’t fit anywhere else in the piece: I think a useful rule to follow is that the only fair criticisms of AI art shouldn’t also perfectly apply to:

• Fan art

• Photoshop

• Photography

• The mechanical reproduction of art prints

• Everyone magically gaining a lot of artistic ability

Before criticizing AI art, just double check that your point wouldn’t also apply to each of these.

You should play around with ChatGPT’s image generator or MidJourney or other image generators if you haven’t yet! I didn’t even get to other forms of AI art like music. I’ve been having a lot of fun generating AI songs using Suno and uploading them here. I’d recommend starting with my Donald Rumsfeld musical. Here’s an album of chill AI background music I put on Spotify.

• Goals for this series

  • Share useful ideas and heuristics

  • Normalize my beliefs in conversations about AI

  • Sharing useful fields to read more about

• Ideas from philosophy I use to think about AI

  • There is nothing fundamentally magical happening in the human brain

  • If physicalism is true, then functionalism is likely true as well—implying that mental states can arise in machines very different from the human brain.

  • Introspection is a bad way to understand our minds

  • To predict the next word, it’s helpful to actually understand the world

  • Words don’t have clear meanings we deeply understand

  • There are probably ways of thinking and experiencing the world so alien to our brains that humans can never access them, but AI could

  • It’s okay to put probabilities on speculative future events, but they often smuggle in a lot of unstated assumptions

  • We shouldn’t rely too much on philosophical arguments to predict specifics about the future

  • Philosophical anti-humanism is underrated