I keep encountering an AI productivity number that has escaped its original date, task and population. It wanders around the internet announcing that the argument is over. Sometimes it says AI makes everyone spectacularly faster. Sometimes it says the opposite. Both versions enjoy leaving the methods section at home.
The latest round of discussion includes Brais Moure's October 8 criticism of uncontrolled AI development. I understand the irritation. Copying faster and producing more code are easy to observe. Whether a team creates more useful, durable work is a harder question.
Consider what METR said when it changed its experiment design in February 2026. Its earlier study found a slowdown for experienced open-source developers using early-2025 tools. The researchers then warned that selection effects and measurement difficulties made their later experiment an unreliable signal of the current effect. They thought newer tools likely helped more, while emphasizing uncertainty about the size. That is a developing research story, not a permanent verdict.
The point is not to pick whichever sentence flatters your current position. The point is to stop pretending a measurement travels through time without losing anything important.
Four things I want beside the number
The task. A first draft, a change to familiar code and debugging a badly described failure are not the same activity. A tool can be wonderful at one and frustrating at another. A broad label such as software development is not enough to explain which work was actually measured.
The person. Someone who knows the system deeply brings different expectations and checking habits from someone learning it. They also spend time differently. An extra minute reading a generated patch might be pointless overhead for one person and a necessary investment for another. Average results do not erase those differences.
The tool and date. A result belongs to the tools and conditions under which it was obtained. We do not normally assess today's compiler by discussing an old version without mentioning the version. Somehow an AI argument can turn a past experiment into a personality test and everyone politely forgets the calendar.
The finish line. Are we timing the first plausible answer, an accepted patch or something that survived use? Did the team count review time? Did it count the abandoned approach? Did it include the person who discovered the awkward regression later? These are choices in what productivity means, not decorative caveats.
There is a second trap. People sometimes hear uncertainty and decide nothing was learned. That is nonsense too. A bounded result can be useful precisely because it tells us where to look next. A result that varies by task can help us choose tasks. A limitation can improve the next experiment. Research does not become worthless because it refuses to behave like an advertising slogan.
At a real desk I would rather ask a smaller question. Can this tool help with this class of change under these constraints? What will the checker need to verify? Where does it repeatedly misunderstand the project? What is the cost of a wrong answer that looks right for the first five minutes?
Those questions are less shareable than a universal percentage. They are also closer to a decision someone can act on.
So put a date stamp on the statistic. Put the task beside it. Leave the uncertainty visible. If doing that ruins the punchline, the punchline was borrowing authority the evidence never offered.
A thought worth keeping?
