The cause was not that the newer model was worse. It was that the newer model spends output tokens differently, and our ceiling had been sized against the old one. Treat a model swap as a change to the system, not a change to a setting.
A model swap is replacing the language model behind a working tool while leaving the prompts, the routing and the interface alone. On paper it is the cheapest upgrade available. In practice it is the one that most often ships broken, because nothing in the code changes and so nothing looks like it needs testing.
What actually went wrong in our run
The Bottleneck Finder is a seven-question diagnostic. A deterministic rules engine decides what the answer should be, and the model writes the prose. That separation is deliberate, and it is why we could swap the model at all without touching the logic.
We pointed it at a newer model and ran the full test set. Fifty-four of ninety-four calls came back with a stop reason of max_tokens, meaning the response had been cut off rather than completed. Briefs ended mid-sentence. We aborted the run and went back to the previous model, which is still what runs in production.
Why does a newer model break a ceiling the old one fit under?
Because the ceiling counts something different now. Anthropic's own model migration guide is explicit about it: on current models, reasoning runs by default, and max_tokens is a hard limit on total output covering both the reasoning and the visible response. The guidance is to revisit that number for any workload that previously ran without reasoning.
Our ceiling had been tuned tightly around the length of brief we wanted. That was correct for the old model and wrong for the new one, where part of the same budget is consumed before a single visible word is produced. The number did not change. What it was buying did.
The general shape of this is worth holding onto. Any value you tuned against a specific model is a dependency on that model, whether or not it looks like one. Token ceilings, timeouts, retry counts, temperature, the length your interface expects back, the format your parser assumes. None of these live in the model, and all of them were set while watching one.
What else moves when the model does?
- Output length and shape. The same prompt produces a different length, which breaks anything downstream that assumed a range.
- Parsing. If you extract structured data from prose, a change in formatting habits breaks the extraction before it breaks the writing.
- Latency and cost per call. Both change, and a tool that was comfortably inside a timeout can stop being so.
- Refusals and edge-case handling. Prompts tuned around an older model's behaviour on hard inputs may not be needed, or may no longer be enough.
How do you test a swap before it reaches anyone?
Run the old model and the new one over the same fixed set of real inputs and compare the outputs side by side. Ours is ninety-four cases, which is enough to surface a systematic failure within a few minutes of reading.
Three things make the test worth running:
- The set never changes between runs. If you change the inputs you cannot attribute the difference to the model.
- Check the stop reason on every call, not the text. A truncated answer often reads fine until the last line. Ours would have passed a skim.
- Count the failures before judging the quality. We never got to an opinion on whether the newer model wrote better briefs, because more than half the run never finished.
This is the same discipline behind the custom AI tools work generally, and it is the reason we keep the decision logic deterministic. When the rules engine owns the decisions, a model swap can only damage the prose, and the prose is something you can read and check.
Where this does not apply
If your tool makes a handful of calls a day, a fixed test set is overhead you do not need. Read the output for a week instead.
A clean swap also does not mean the new model is better for your job. It means it did not break. Those are separate questions and the second one takes longer to answer, because it needs a human reading output against a standard, not a pass rate.
And this post is one run on one tool. A different workload with a generous token ceiling may well swap with nothing happening at all. The lesson is not that newer models truncate. It is that a value you tuned against one model is not guaranteed to survive the next one.
A one-line change still deserves a test run, because the line is not the size of the change.