The smallest model in the comparison is an elderly bicycle. The cloud model is a new motorbike. The reviewer announces that wheels have been settled forever. Everyone in the comments then starts arguing about whether the bicycle community is lying.
That is a metaphor, before anyone attempts to benchmark transport with a language model. But it captures the irritation in an October 7 LocalLLaMA comment thread: commenters disputed the use of older local models in a comparison. The original post is gone, so I am not going to reconstruct its full experiment from angry replies. The surviving disagreement is enough to raise the methodological question.
What are you actually comparing?
A current local model against a current hosted system? A system someone already deployed against a possible replacement? Two options that fit the same memory budget? A cheap tool against a more expensive one? Each could be a useful comparison. None becomes useless simply because another comparison might produce a different winner.
The trouble starts when the question changes after the result arrives.
If you test an old deployed system because that is what a particular team uses, say so. The finding might help that team decide whether to update. It does not establish the limits of every model that can run locally. If you test the newest available options, describe the access, configuration and task. That does not automatically tell an organization what it can responsibly deploy tomorrow.
Let the losing result stay in the notebook
I want the awkward cases. The instruction the favorite model misunderstood. The answer that looked elegant until somebody checked the requirement. The run where the cheaper option did enough. The run where it clearly did not. A notebook that contains only victories is closer to a fan page than an experiment.
This also means separating an explanation from an excuse. A different prompt might improve a result; that is a new test. A different setting might be more appropriate; record it and test again. Announcing that the model would certainly have succeeded under an imagined perfect setup is not a result. It is a story about the result you wanted.
Equally, a failure deserves a precise description. Did the answer use the wrong fact? Miss a constraint? Fail to ask a question? Produce valid-looking output that did not meet the task? Those are more informative observations than a verdict that the model is stupid. They tell another person what to inspect.
I am especially tired of comparisons that make the audience choose a camp before explaining the workload. The cloud camp and the local camp can each find impressive demonstrations. Neither camp has a monopoly on convenient omissions. Privacy preferences do not improve an incorrect answer. A strong answer does not settle who should receive a confidential document.
You can admire a model and still ask for a fair experiment. You can want local control and admit that a hosted option did better at a particular task. This is not a loyalty test. It is the basic courtesy of allowing the question to survive its answer.
My preferred comparison ends with a narrower conclusion than it began with. This option worked for these tasks. That one failed in these ways. Here is what we did not test. Here is the next thing worth checking.
If your benchmark ends by announcing that everyone outside your favorite community is an idiot, I suspect the social experiment was always the main attraction.
A thought worth keeping?
