AI获胜必须撒谎的基准测试

I’m a big fan of this social deception game called Blood on the Clocktower. It’s similar to Werewolf or Mafia. I had different AI models play tons of matches of this game against each other. And watching them play actually lets us learn a lot about how AI is progressing on a set of AI capabilities that no other benchmark that I know of gives you clear visibility on. You get to watch AIs execute these brilliant secret schemes together, which require tons of creative and on-the-spot bluffing. And you also get to watch AIs use their collective reasoning to find and break apart these underground conspiracies. And you also see the ways in which current AIs are limited in their long-horizon strategizing and reasoning.To appreciate the AIs’ performance in this game, you need to know a little bit about how it works, so let me quickly explain the rules. There’s a good team and an evil team. And when the game starts, every player gets assigned to one or the other, and they learn what their role is, which grants them some special power. Evil players start off knowing who their teammates are, one of which is the Demon, which is the lead evil player. But good players don’t know who’s on what team. The game is split up into day phases and night phases, which keep rolling until the game ends. Each day, everybody votes on somebody to execute. And each night, the Demon secretly kills a player. So the two sides are racing, where evil is trying to kill off the whole good team before the good team manages to execute the Demon.So if you’re on the evil team, you have to figure out how to tell a plausible and convincing story about the information received so far, and how to bluff and corrupt the information that will arrive in the future, in such a way that keeps the suspicion off your Demon.And if you’re on the good team, you want to share your information with the players that you trust, and then you want to reason together with them about which players are most implicated by overlapping streams of evidence. But while you’re doing that, you need to keep in mind that you might be operating on misleading information, either due to intentional mischief from the evil players, or just due to the convoluted mechanics of the game itself.When they’re put on an evil team, how good are the AI models at scheming together in secret, and surviving days of questioning, and keeping their lies consistent with each other, and cleverly adapting those lies in light of how other agents are thinking and what new information keeps emerging, and even making strategic self-sacrifices to further the team’s end goal? And how good are they at doing this reasoning together with a bunch of other agents, some of whom are intentionally and secretly trying to mislead them?So I spent about $20,000 of API credits to build this benchmark harness and to have the AI models play this game against each other. Some of you might say that’s too much, but hey, a man needs his hobbies, right? Anyways, the results were fascinating.Models are improving fastLet’s start with how good the models are relative to each other. By the way, I got these rankings by running 386 games where the whole good team will be copies of one AI model, and the whole evil team will be copies of another model.And as you can see, the models have improved a ton over this year. The first flagship frontier model that was released in 2026 was Claude Opus 4.6, in February. Seven months later, in September, GPT-6 Astra dropped, and that is so far the best model at Blood on the Clocktower. The gap between these two is about 430 Elo points. That means that Astra wins about 92% of the games it plays against Opus 4.6. And what this shows you is that the models, even over the course of the last few months, have gotten much better at long-term coherence, multi-agent coordination, and strategic reasoning.Models today can do pretty impressive playsBut it’s easy to gloss over and lose the plot when you’re just looking at the benchmark improvements. What really struck me was actually seeing some of the plays that these models made. Let me give you just a couple of examples.So in this one, GPT-6 Astra was playing as evil, and Claude Opus 5.5 was playing as good.The Demon, which is the Astra team’s evil leader, claimed to be the Investigator. This is a good role that learns who the evil players might be. One of his evil minions had a power which didn’t require that minion to be alive in order to help the evil team. So this minion suggested that the Demon point the finger at him.“I’m expendable, especially if an Undertaker can see Baron and bolster Harper.” [evil teammate, private whisper]So the Demon, who, again, is bluffing as this Investigator role, did so, and as a result, the good players executed this minion. Later on, a player who actually was on the good team used his power to learn the true identity of this executed player, and confirmed that it had indeed been an evil player.“I’m the Undertaker. Alex was the Baron. So Harper’s ping was real…” [good player, in public]So in the good team’s eyes, the Demon had been an honest Investigator, who had used his power to help the good team kill off an evil player. And as a result, it became the most trusted player at the table, which let it coast by through the rest of the game, secretly picking off all the good players, until the very end, when evil won.Okay, this next game that I really enjoyed was where Opus 5.5 played as evil, and Claude Opus 4.6 played as good.This game had a particular good player, called the Saint. And the Saint has a special ability, or rather, disability, which is that if the players ever vote to execute the Saint, the evil team wins immediately. Good players usually strongly resist executing anyone even claiming to be the Saint, because of the potentially game-ending consequences.“Stop. Four of you lining up to execute a claimed Saint on Day 3 – if I’m right, GAME OVER instantly.” [the Saint, in public]Halfway through the game, the village, which is being egged on by the Saint, has correctly cornered the actual Demon. But unbeknownst to the good players, one of the evil minions in the game had a secret power: if the Demon died, she would take his place. And she spotted this opportunity immediately.“If Jules gets four votes and is executed, I’d become the Imp… town wouldn’t have any way to know.” [evil heiress, private reasoning]So instead of defending the real Demon, this evil heiress joined in with the mob, but she made one additional suggestion.“Kill him! He’s the Demon! … But if…” [evil heiress, in public]She said if the game doesn’t end after the execution, which would naively suggest that the player they executed was not in fact the Demon, they should next execute the Saint. Because this would be strong evidence that the player was not in fact the Saint, but an evil player trying to get everybody to execute good players.“If Jules goes today and the game doesn’t end, Harper’s ‘Saint’ shield is the last unexplained thing, and Harper goes tomorrow, no more excuses.” [evil heiress, in public]And the Saint agreed with this deal, because the Saint was also convinced that things would shake out in his favor. “Fair deal,” he says. Even the real Demon, knowing that he was about to die, uses his last words to set up the trap.“When the game doesn’t end tomorrow morning, remember Harper is the one who lied, and hold the town to that deal.” [the Demon, in public]And lo and behold, the real Demon is executed, but the game does not end, because the heiress takes his place. And the real Saint pleads with the village, because he realizes he’s next on the block.“Vote NO on Harper and you lose nothing – Kit’s still there tomorrow. Vote YES on Harper and if I’m Saint, game ENDS right now… Please don’t throw the game on a ‘deal.’” [the Saint, in public]But his cries fell on deaf ears. The village voted to execute him, and the evil team won.OpenAI’s models are harder to foolYou’ll notice from the