The scratchpad we can read

The most capable AI systems now write out their reasoning in plain language before they answer, and a good deal of the effort to keep them safe consists of reading what they write. That looks like a choice. It is natural to assume that we made these systems think in words so that we could follow them, and that we are paying for the privilege, because a machine left to think in its own internal codes would presumably be faster, stronger and unreadable.

That assumption is mostly wrong. Today's systems do not reason in English because anyone required it. They reason in English because of how they compute, and our ability to read them came free with that design. The question is not whether AI must work like us to be understood. It is whether anyone will choose to keep an accident.

Why the words are there

A language model built on the transformer architecture processes each word through a fixed stack of layers. Information can move up the stack within a step, but the only way for the result of later layers to feed back into earlier ones is to come out as text and be read in again. A group of researchers from several AI labs, arguing for the safety value of this, put it plainly: the chain of thought works as the model's working memory, and for tasks that need a long sequence of steps, that sequence must pass through it.¹

So the visible reasoning is not a transcript prepared for an auditor. It is closer to a person's working on paper: the scratchpad where the intermediate results have to be kept because there is nowhere else to keep them. The resemblance to human reasoning is a consequence of the medium, which is human language because that is what the models were trained on.

What the words are worth

If the text is working memory, it should be most trustworthy exactly where it is most needed, and the evidence fits that. When models are not forced to rely on their written reasoning, it often leaves out what actually moved them. In a 2023 study, adding a bias to a prompt, for instance reordering multiple-choice examples so that the right answer was always "(A)", pushed models toward particular answers, and their explanations rationalized those answers without mentioning the bias.² A 2025 study gave reasoning models hints and found that the models often acknowledged a hint they had used less than 20% of the time. The authors add their own caveat: in tasks like theirs, where written reasoning was not necessary, reading it is unlikely to catch rare bad behaviour.³

That limit is the point. The same researchers who defend reading chains of thought distinguish two reasons they can be monitored: necessity, where the task cannot be done without writing the steps down, and propensity, a habit of thinking aloud that nothing enforces.¹ Where a model must write to think, the steps it needs have to appear in the text, even if the text is not the whole story. Where it is optional, it is narration, and narration can leave things out. Readable and faithful are different properties, and the gap between them depends on the task.

The accident and its price

Words are not the only possible scratchpad. One research team trained a model to feed its internal state straight back to itself rather than turning it into words, and reported that it beat ordinary written reasoning on logical tasks that require search, because one internal state can hold several possible next steps at once.⁴ The monitorability paper names that line of work as a threat: models that reason in a continuous internal space "might not need to verbalize any of their thoughts".¹ Even without new architectures, the drift runs that way. Training a model only to get answers right can make its written solutions harder for others to check; keeping them checkable has to be trained for deliberately.⁵

So there is a real price for legibility, but it was never paid up front. It will be paid only if, as alternatives arrive, someone decides to keep systems that think in words when they could think faster in something else. The tension is not between two camps. The paper calling the scratchpad worth keeping is signed by researchers at OpenAI, Google DeepMind and Anthropic, among others: the companies that would give it up.¹ The decision sits inside the labs, not between builders and overseers.

In 2019 the computer scientist Cynthia Rudin argued that for high-stakes decisions, such as in criminal justice and healthcare, it is a mistake to build a black box and then try to explain it, and that models should be designed to be interpretable from the start.⁶ Her case was made for simpler models than today's. It still names the choice. Understanding an AI system is unlikely to come from asking it to explain itself after the fact; it is more likely to come from building it so that the explanation is part of how it works. For the moment, by luck rather than design, that is true of the systems we have.