致史蒂芬·平克的一封公开信

Dear Steven:Earlier this month, frustrated by some of your social media posts, I challenged you to a public debate on AI.I hadn’t realized that debate offers are to New Media what chum is to sharks. My inbox was deluged by podcasters, debating societies, and general influencers, all offering to host. When I objected that maybe we should wait for you to accept before figuring out details, they all said great, that’s absolutely right, they would contact you right away. Their people would reach out to your people. Their nephew knew your friend’s babysitter and they were confident they could relay the message. They had already rented an empty lot near your house where they would place a speaker blaring “RESPOND TO SCOTT ALEXANDER’S DEBATE OFFER!” at max volume, day and night. I protested that probably asking you directly on Twitter was enough to get your attention, and maybe now we should give you some time to make up your mind before pressing further, but they said that no, sorry, they’d already hired the team from Inception to place subtle debate-related messages in your dreams. So I feel a little bad for unintentionally harassing you, and doubly so after you responded in your trademark reasonable and charitable style with An Open Letter To Scott Alexander. You write that in-person debates seem confrontational and not truth-seeking. Apologizing for any inadvertent offense you may have given with your social media posts, you nevertheless deny my accusations of personal attacks, saying that your primary contribution to the discourse has been reasonable arguments about why AI might not be so dangerous after all (which you go on to list). You invite me to respond to these arguments in the same spirit of respectful mutual intellectual inquiry with which they are offered.I agree that in-person debates are confrontational and bad for truth-seeking. But I didn’t propose a debate in order to seek truth. I proposed it because, under California Penal Code § 415(1), it’s illegal for me to challenge you to a duel. I think your public writing on this topic has been dishonorable. Out of obligation, I will respond to the meaty arguments that you have set out for me. But what would be viscerally satisfying would be to make you get up on a stage where I read your own words to you in real time and ask “Really? Really?” after each sentence. Then I could watch you squirm as you try to square your output with your status as one of America’s top public intellectuals.So I’ll start by addressing the meaty arguments, move on to directly impugning your honor, and then we’ll decide whether to debate, duel, or seek some kind of synthesis of our conflicting views.Four Arguments Against SuperintelligenceTaking the four arguments in your open letter in order:1: [Doomer scenarios have] an underdeveloped conception of intelligence which treated it as a quantity of power which may be extrapolated from animals to dull humans to smart humans to AI to Artificial Superintelligence, the latter consisting of perfect omniscience. (This was the focus of my later exchange with Scott Aaronson.) I argued that any intelligent system is a mechanism which is good at solving some problems but not others, and which is inherently limited by knowledge about the world attainable only by observation and experimentation.GPT-6 is more intelligent than GPT-3. You can split hairs, you can come up with some obscure sense in which that word is slightly off, but common-sensically it’s true. In the same way, GPT-9 will be more intelligent than GPT-6. At some point, this intelligence will exceed human intelligence. My argument requires nothing more complicated or philosophically dubious than this.I feel like we’ve had this discussion before. At each GPT level, you seem to have believed it was implausible that AI would get more intelligent than it was already. Credit to you for starting early - you began predicting this in 2019, around GPT-2:In 2022, we were part of a sprawling many-sided debate with Scott Aaronson and Gary Marcus. Marcus had noticed that GPT-2 and GPT-3 couldn’t solve basic math problems, like “I put two trophies on a table, then add another - the total number of trophies is now ______?” I argued that as AI scaled up, GPT-4, GPT-5 and so on would avoid these failure modes and become smarter. You, sticking to your claim that intelligence was not a meaningful lens through which to think about the problem, guessed that it wouldn’t:Of course, GPT-4 was able to add 2+1 just fine, GPT-5 could do harder problems still, and an internal OpenAI model just beyond GPT-6 recently solved Navier-Stokes.In 2023, you tripled down on the claim, noting that the AIs of the day still couldn’t figure out that someone alive in the morning and evening was alive in the afternoon, and saying that “I doubt it will improve exponentially”:The trajectory of AI improvement since 2023.I think at this point we should acknowledge that the default hypothesis is that each GPT generation will be better than the last, and that - while this will no doubt asymptote eventually - we have no compelling argument for placing the asymptote at any particular point, including “right here” or “just before the AI reaches human level”.Does this require reifying intelligence as a specific mysterious fluid? You have ably written about how this seemingly absurd claim holds for humans - specifically, all intellectual abilities are correlated in a mysterious pattern called g, the basis of IQ tests. But maybe this is only a quirk of humans, which cannot be extended to other minds? Psychologists’ two favorite things are rats and IQ tests, so they have hardly failed to give IQ tests to rats. They’ve found have a g-like structure linking all of rats’ cognitive abilities too. Reader, Hager, and Laland did the work in primates, MacLean et al extended something similar to birds, and more recent work compares all animals to one another. All of these studies confirmed the common-sense view that there is a general factor of cognition across animal species (ie chimpanzees really are smarter than mice or ants), and it maybe be as basic as a simple function of neuron number.Maybe this is sensible in biology, but not artificial intelligence? No - comparing AIs across generations (eg GPT-2 to GPT-6), we find that the AIs that are best at translating languages are also the AIs that are best at writing sonnets and superforecasting political events, often without specific attempts to train these additional skills. And the same AIs that are better at all these skills are also better at “fuzzy” skills like situational awareness, agency, executive function, and long time horizons. In case you need a study to prove this obvious thing, Ruan, Maddison, and Hashimoto find that a single factor (general intelligence) explains about 80% of the variance in an AI’s scores on any benchmark, ranging from grammar to trivia knowledge to software engineering. Epoch has adapted something like this into its famous Epoch Capabilities Index, a sort of IQ-type number for AI which closely matches people’s subjective impression of their abilities. Here’s a graph of how ECI has been changing over time:For me, the most parsimonious explanation is that AI intelligence is a function of scaling laws (parameter count and training data) in a way that’s pretty close to how animal intelligence is a function of neuron number. If we truly understood the domain, maybe we could have a single scaling law relating animal neurons to artificial neurons which - plus or minus a term for training/experience - could explain the intelligence of both types of entity. At least, this is what I told Gary Marcus in 2022, and it did a pretty good job of predicting what happened between then and now.But I don’t need you to accept my speculative philosophical research program - only the drop-dead obvious fact that GPT-6 is better than GPT-2 in every way, and the same process that produced these improvements will soon be u