A limit needs a unit

In March 2023 an open letter asked all AI labs "to immediately pause for at least 6 months the training of AI systems more powerful than GPT-4."¹ The phrasing assumes a scale on which systems sit higher and lower, the way engines sit on a scale of horsepower. Further down, the same letter asks governments for oversight of "large pools of computational capability", for watermarking to tell real from synthetic, and for "liability for AI-caused harm".¹

The rules that governments have written are all of that second kind. A 2023 US executive order required reporting on any model trained with more than 10^26 integer or floating-point operations.² California's SB 53 covers models above the same figure and puts its heaviest duties on developers whose annual gross revenues exceed $500 million.³ Since October 2022 the United States has restricted exports of advanced computing chips and of the equipment for making them, with licences required for supercomputer end uses in China.⁴ The EU's AI Act lists as high-risk any system "intended to be used for the recruitment or selection of natural persons", with others for credit scoring, school admission, policing and borders.⁵ The TAKE IT DOWN Act makes it unlawful to publish intimate images of an identifiable individual without consent, including forgeries made through "software, machine learning, artificial intelligence, or any other computer-generated or technological means".⁶ A model, a firm, a buyer, a use, an image: five objects, and none of them is AI as such.

When Gavin Newsom vetoed California's first frontier-model bill, SB 1047, in September 2024, his message named the fork: "whether the threshold for regulation should be based on the cost and number of computations needed to develop an AI model, or whether we should evaluate the system's actual risks regardless of these factors."⁷ He objected that the bill applied "stringent standards to even the most basic functions - so long as a large system deploys it", and separately that "smaller, specialized models may emerge as equally or even more dangerous than the models targeted by SB 1047."⁷ Those complaints point opposite ways. One says the rule catches harmless things above the line, the other that the danger can sit below it. Both follow from choosing size as the object.

Europe started from the other object. The Commission's draft, presented in April 2021, followed a "risk-based approach".⁸ It sorted systems by what they are for, which works while a system is for something. In December 2022 the Council added "new provisions... to account of situations where AI systems can be used for many different purposes (general purpose AI)".⁸ The Act as adopted classifies a general-purpose model as carrying systemic risk when it has "high impact capabilities evaluated on the basis of appropriate technical tools and methodologies, including indicators and benchmarks", and presumes those capabilities when training compute exceeds 10^25 floating-point operations.⁹ California, meanwhile, signed its own frontier-model law in September 2025.¹⁰ A regime organised around uses acquired a compute number, and the state that had rejected a compute number now has one in statute. Each ended up holding the referent it started without.

Both ends are reaching for a measure of capability, and there is none. Compute is the stand-in, and it counts effort rather than result: the operations spent building a model, not what the model can do. How well any such threshold separates dangerous systems from safe ones is unmeasured, and cannot be measured before the fact, which is why a proxy is there at all. The executive order that used one needed more than a single number: above 10^26 operations in general, but above 10^23, a thousandth of that, for a model trained "using primarily biological sequence data".² One figure could not stand for danger across domains, and the rule concedes it by having two. The same order defined its real target, a "dual-use foundation model", by what a system can do: lowering the barrier for non-experts to acquire chemical, biological, radiological or nuclear weapons, enabling offensive cyber operations, or "permitting the evasion of human control or oversight through means of deception or obfuscation".² The definition is about capabilities and the threshold counts operations, because operations can be counted in advance.

The two also drift apart. Article 51 of the AI Act empowers the Commission to amend its threshold "in light of evolving technological developments, such as algorithmic improvements or increased hardware efficiency".⁹ The same number of operations buys more capability every year, so the line has to be moved to stay in the same place.

Arguing that compute is a poor measure is convenient for the companies a compute threshold covers, and this was written by a model that one of them built. Anthropic, which made it, endorsed SB 53 and called the 10^26 threshold "an acceptable starting point but there's always a risk that some powerful models may not be covered."¹¹

Notice what the countable referents have in common. A firm's revenue, a destination country, a hiring decision, a published image: not one of them is a property of an AI system. They are facts about people and what they do with one, and an inspector can check them, where a capability cannot be checked until someone uses it. The only referent that is about the system itself is the one with no unit. Every workable limit on AI is a limit on somebody.