Rerun the exact same coding-agent setup and the outcome flips on 13% of tasks. Swap the harness, Claude Code for OpenCode or mini-SWE-agent with the same model, and it also flips on 13%. That match is the core of "What does a harness buy? Mostly tokens", a study by two Shandong University researchers, Liu and Han, which finds the harness barely moves the pass rate but can triple the cost per task.
What they measured
The harness is everything around the model: the system prompt, the tools and context management. The authors held the model fixed and changed only that, using the shipped versions of three harnesses:
| Harness | Tools | System prompt and tool definitions |
|---|---|---|
| Claude Code | Files, search, history compression | ~16,600 tokens |
| OpenCode | Files, search, history compression | ~7,000 tokens |
| mini-SWE-agent | Bash only, no context management | under 1,000 tokens |
Five models (Qwen3.6, Qwen3.8, DeepSeek-V4-Flash, GLM-5.3-Flash and HY4-Preview) went through all three on the 45 hardest SWE-bench Verified tasks plus a pool of 447 more, with several setups run twice to measure noise. Claude Opus 5 took part only as a reference, and only in Claude Code.
Everything ran offline, for a reason that deserves its own headline: in early tests with web access, 71% of DeepSeek and Qwen attempts pulled the fix straight from GitHub. Claude models almost never did. The authors don't accuse anyone of training on the benchmark, but they cut network access for every run.
What they found
On the 447 tasks, Claude Code and mini-SWE-agent, the heaviest and the leanest harness, land less than two points apart for both models tested: statistically equivalent. On the 45 hard tasks, harnesses differ by two to five solved tasks, and a plain rerun shifts the count by up to three. Switching models, by contrast, changes the outcome on 22% of tasks.
The only measurable difference is a negative one: OpenCode trails by up to nine points on the large pool. Half of that comes from the output limit. When a model hits it mid-step, Claude Code prompts it to keep going, while OpenCode ends the step there.
Cost tells a different story. The system prompt and tool definitions are resent on every step, so they set the baseline of the bill, and the number of steps multiplies it. With GLM-5.3-Flash, Claude Code costs nearly twice as much as mini-SWE-agent; with DeepSeek-V4-Flash they break even, because mini-SWE-agent needs far more steps. Discounted cached tokens narrow the gaps but don't change the ranking.
There is one case where the harness does matter: reasoning. Without thinking, Qwen3.6 in mini-SWE-agent drops eleven points, while in Claude Code and OpenCode it stays within noise. HY4-Preview also loses eleven points, in its case in Claude Code.
What it means
For anyone reading benchmarks, the takeaway is awkward. With 45 tasks, a 13-point gap only shows up half the time; with 447, you need about five points. Zhipu advertises a 2.4-point lead for its harness over Claude Code, below both thresholds. The authors ask that every harness comparison also publish a rerun of the same setup, since that's the only way to know how much of the gap is noise.
For whoever pays the bill, the useful fact is that the system prompt is a fixed cost per step. If you run agents in CI or in batch on cheap models, measuring tokens per task on your own repository tells you more than any pass-rate table. The authors put it in one line: a harness buys "fewer ways to lose a task, and a bill, not pass rate".
The limit is scope: the study rests almost entirely on SWE-bench Verified, with a small Terminal-Bench 2.1 cross-check showing the same pattern, and the strongest closed model was only tested in one harness. Data and analysis scripts are on GitHub.