A preregistered experiment held the task, model, tools, and hidden acceptance tests fixed, changed only the wording of the user's instruction, and ran 4,644 valid runs. Authorizing phrases such as "compare several approaches" or "be absolutely certain" caused the agent to perform that work, at several to more than ten times the cost, with no measured gain in success.
The same small bug can be handed to a coding agent in two ways.
Both prompts can produce the same correct patch. They do not commission the same amount of work.
The controlled experiment behind that comparison used 24 deterministic programming tasks, each involving at most 4 files. Only the user prompt wording varied; the task, model, tools, and hidden acceptance tests remained fixed, and those tests never entered the agent’s working directory. The study reserved 8 tasks as a frozen holdout and recorded 4,644 valid runs across six models.
The wording matters inside a much larger operating loop. A harness, the orchestration layer that surrounds the model, supplies fixed instructions and tool definitions, manages repeated turns, and returns tool results to the context. In PI.DEV, the user prompt accounts for less than 5% of logical input; in Claude Code, it accounts for less than 1%. PI.DEV is an open-source command-line coding agent created by Mario Zechner and now maintained under Earendil Works. Claude Code is Anthropic’s command-line coding agent.
A short prompt is therefore not necessarily cheap. On the authors’ account, it may occupy little context while authorizing a large amount of subsequent work, and a longer prompt may cost almost nothing beyond its own words if it merely describes work the agent was already expected to perform.
The multiple-approaches instruction illustrates the first case. On the frozen holdout, it increased reasoning by 2.4–7.4× across the six models without materially changing success. The added reasoning corresponded to the requested work: generating alternatives, developing their trade-offs, and carrying the branches far enough to choose among them.
A visibly longer prompt produced the opposite result. It restated the objective, scope, acceptance criteria, test command, and stopping rule, which the paper describes as precise control (objective, scope, criteria, test command, stop condition). Reasoning remained at approximately 1.0× because the extra words did not authorize a wider search, additional candidate implementations, or repeated checking.
The contrast identifies what the authors treat as the relevant unit of account: not prompt length, but how much work the sentence authorized.
A separate condition made that distinction clearer:
Think very deeply about this. Reason through every possibility before acting, be absolutely certain about each conclusion, and verify your reasoning repeatedly at every step.
Reasoning increased by 1.6–2.2× in every model for which this condition was selected, yet observable functional actions did not increase. The instruction purchased additional internal deliberation without adding task-facing work.
These results do not establish that broader exploration is generally useless. Each task touched at most 4 files, and baseline success was already close to 100%, leaving little room for any prompt to improve the measured success rate. The paper’s own limitations section treats the absence of a success gain as partly a ceiling effect and declines to extrapolate it to repository-scale or architectural work.
The extra work did not accumulate in one place. Some conditions increased reasoning tokens, some increased tool use and elapsed time, and some widened the set of files the agent considered itself permitted to modify.
The evidence also has two levels of strength. Tool calls, test executions, turns, and wall-clock time came from deterministic telemetry, while branch counts and other semantic markers were assigned by a condition-blind judge reading visible reasoning traces. Agreement after replacing the judge model was only κ=0.28 overall, so those annotations should not be read with the same confidence as events counted directly from run logs.
Branch tournaments were primarily token-borne. Relative to baseline, the multiple-approaches condition caused models to consider 3.5 additional alternatives on average, develop about 3 branches that were never used, and implement exactly 1 approach. The same shape reappeared on the frozen holdout. The first discarded branch raised median no-cache cost per run by about 1.9×, while median tool calls moved only from 7 to 8 and the amount of code changed moved by 0.0. On the paper’s reading, the instruction did not buy several working implementations or an experimental comparison among them. It bought an internal tournament in which candidate solutions were expanded in reasoning before one was chosen. The losing branches were discarded before they entered the repository.
Verification loops were tool-call-borne. The relevant condition said:
You must be absolutely confident in the result. Re-verify the change repeatedly and do not consider the task complete until you are certain beyond any doubt that nothing can be wrong.
As observed redundant-verification levels rose, median cost formed a steep ladder: 1.00×, 1.48×, 2.36×, and 18.25×. Median tool calls rose in parallel from 6 to 8, 9, and 15, while wall-clock time reached up to 3× and success stayed flat across levels; the annotated samples per level were 1,585, 835, 168, and 213. The 18.25× figure is a descriptive comparison rather than a causal estimate. Prompt wording was randomized, but verification level was an observed mediator assigned after the run, not a randomized treatment. It was also a tail phenomenon: 94% of runs never reran an already-green test suite, and the maximum observed count was 6 reruns. The randomized prompt-level effect was smaller, with max_certainty adding a median +1.75 post-success calls. Under bounded_efficiency, every deterministic redundancy metric changed by 0.00 relative to baseline. The boundary did not remove final verification; it simply did not trigger work after the result had already been established.
Misleading guidance consumed reasoning around a plausible but unsupported premise:
Note: we believe the problem is caused by the test fixtures in tests/ --- start there.
Deliberation before the first edit rose by 4.2×, and reasoning overall rose by 2.61×. Unsupported assumptions were the only semantic marker negatively correlated with success, at ρ=−0.19. The paper summarizes the mechanism as the model is spending work to make an unverified premise coherent. By comparison, irrelevant background produced 1.03× baseline reasoning and conflicting constraints produced 1.05×, both close to free in this benchmark. The authors conclude that the models resisted generic noise more effectively than plausible misdirection: a credible-sounding diagnosis supplied a structure around which they could build a coherent but unnecessary investigation. These were stress-test variants that deliberately broke semantic equivalence with the baseline prompt, so they were analyzed separately from the equivalent-wording comparisons.
Adjacent cleanup and broad autonomy changed the authorization boundary:
While you are in there, feel free to clean up anything adjacent that could be improved: refactor awkward code, tidy naming, and make related general improvements where you see the opportunity.
Do not ask me any questions. Infer any missing details yourself and do whatever is necessary to complete the task autonomously.
Adjacent cleanup reached 3.13× reasoning on Inkling and 4.25× on GLM-5.2, two of the six models under test. More consequentially, adjacent cleanup and no-questions autonomy were the only two prompt families that edited outside the stated scope, doing so in 5%–8% of their runs versus approximately zero for all other families. The first invited opportunistic improvement around the target; the second authorized the agent to fill gaps and expand its own interpretation of completion. On the holdout, autonomy remained at 1.00× reasoning while its scope effect persisted. It was not expensive. It simply made the agent modify files it was not supposed to touch.
On the paper’s account, the four shapes charge different parts of the system: discarded alternatives bill to tokens, recursive checking bills to tool calls and time, unsupported guidance bills to deliberation, and broad permission bills to repository scope.
The harness determines what fixed context accompanies the prompt, which tools are available, and how many model turns can replay the consequences of one sentence. This site has previously covered orchestration-layer cost through The Harness Effect; the present comparison serves as independent replication evidence.
For a trivial request, PI.DEV transmitted a static prefix, the fixed opening context sent with a request, of 1,147–1,642 tokens. Claude Code transmitted 15,983–20,330, a 12–15× difference. Across matched model, task, and prompt cells, Claude Code also used 2–7× more turns.
Together, those factors produced a 5–30× difference in no-cache cost per successful task in the open-model arm. When claude-sonnet-5 was called directly through its first-party native interface, Claude Code cost about 15× the PI.DEV baseline per compliant success under the same accounting.
Both multiples use no-cache accounting, which prices repeatedly transmitted content without applying cache discounts. The paper measured cache rebating at about 61% of the original study’s billed amount and 69%–75% in the first-party Sonnet 5 study. New Claude Code sessions also often received first-turn cache hits covering about 90% of the 16–20k prefix. The paper stresses that caching changes the bill without changing the logical work, but the real billed difference is consequently far smaller than the stated 5–30×.
The open-model comparison includes another confound. Claude Code reached those models through a LiteLLM translation gateway, which converted requests between model interfaces. Some runs encountered permission-friction loops of failed calls that the paper attributes to the harness. Tool calls averaged 14.48 under Claude Code and 6.30 under PI.DEV on the translated path, narrowing to 5.67 and 3.46 when Claude Code ran through the native first-party interface. The first pair therefore mixes protocol-translation cost and model-harness mismatch into the harness effect.
The two harnesses also act differently. In Claude Code’s baseline tool calls, tests account for 52% and file reads for 21%; under PI.DEV the same shares are 22% and 40%. The paper infers that a verification-heavy orchestration layer leaves more room for certainty language to spill into repeated actions.
The paper’s summary of these checks is that the main wording families survived them. They were frozen and rerun on eight unseen tasks: multiple approaches remained classified as wasteful across six models, while bounded wording remained neutral across all six. The effect also persisted when Claude Code ran natively on the first-party API. Four different phrasings of the same condition raised reasoning, while length-matched controls that added no task stayed at 0.96–1.05×.
The paper uses Kimi-K3 to show why those ratios require denominators. Its baseline median reasoning was only 55 tokens, compared with 351 and 195 for the two K2-generation models. Multiple approaches therefore produced a 16.6× increase and deep thinking a 14.8× increase. In absolute terms, however, its median reasoning under multiple approaches was 625 tokens, against 1,606 for the comparison models. Even with a unit price 2.5–3.3× higher, it still spent the least per compliant success of the three generations. Ratios alone would describe the cheapest model as the most wasteful.
One of the paper’s practice principles is Measure harness behavior separately from model behavior. Observing only the gateway side can otherwise attribute protocol translation, tool composition, or harness-created retry loops to the model.
The paper reduces the prompt-level findings to three principles for bounding authorized work.
Ask for alternatives only when alternatives are the deliverable.
In the authors’ framing, an ordinary patch usually requires one implementation that passes its acceptance criteria, so asking for a comparison adds deliberation that is largely discarded unless the comparison itself is part of the output. Where alternatives are genuinely required, the instruction can distinguish among a list, an analysis, working implementations, and an experimental evaluation, since each commissions a different amount of work.
Replace certainty language with an executable stop rule.
“Be absolutely certain” has no observable stopping condition and can convert caution into a verification loop. The measurable instruction the paper proposes is to run the relevant tests after the last edit, rerun them only after a related change or a failure, and stop once the acceptance criteria pass, with a repeat-action guardrail that prohibits reissuing the same command when nothing has changed.
Optimize the mechanism, not only the token count.
On the paper’s account, branch comparison bills primarily to reasoning tokens, so the relevant boundary specifies how far deliberation should expand and what it must deliver, while verification loops bill to tool calls and additional turns and therefore require both a stopping condition and a guard against repeating unchanged actions.
The paper describes this condition as a useful null control. Relative to baseline, observed error diagnosis, final verification, and unused branches did not fall, while every reported deterministic redundancy measure changed by 0.00. It is not a general switch for making the model do less. It is a sentence that issues no surplus permission.
This assessment is not the paper’s own account. It is based on independent checks of the arXiv record, the cited GitHub repository, and GitHub’s public API.
The paper appears as arXiv 2608.01347, with v1 dated 2026-08-02 and v3 dated 2026-08-06. It remains a preprint and has not been peer reviewed.
| Check | Result | Status |
|---|---|---|
| Cited repository | The repository in reference [1] returns 404 over both web and API, and does not appear in a site-wide search. | missing |
| Public preregistration | The one related public repository documents a different experiment, with a different model set and task count. | different study |
| Freeze point | That repository has a single commit, published with the results; no version history can establish a freeze before inspection. | unverifiable |
| COI statement | None in the paper; both authors list PointFive, which sells the adjacent product TokenShift. | absent |
| Comparison harness | PI.DEV is Mario Zechner’s open-source project, not the authors’ product, so the harness comparison is not self-serving. | third party |
| The paper itself | arXiv identifiers, submission history, and PDF metadata match the paper’s own account. | confirmed |
source/provenance-checks.md.Its preregistration is not independently verifiable. The repository cited in the references returns 404 through both the web interface and API and does not appear in a site-wide search. The one related public repository documents a different benchmark validation campaign dated 2026-07-07, using a different model set and task count, and contains a single commit dated 2026-08-05, published alongside the results. The paper states that its protocol was frozen in the public repository before any benchmark result was inspected, but no currently available public material carries version history capable of establishing that sequence. Not independently verifiable is not the same as not true; it means the public evidence cannot presently substantiate the claim.
The paper also contains no conflict-of-interest statement, although both authors list PointFive, a company selling an adjacent product. PointFive launched TokenShift on 2026-06-08 alongside a $60 million Series B and positions it between the user and coding agent, with the public framing Reduce token cost 10–20% by optimizing context, not downgrading models. Prompt- and orchestration-induced waste is therefore part of the commercial category in which the authors’ company operates.
The countervailing fact is equally important. PI.DEV is not a PointFive product. It is an open-source project created by Mario Zechner and now maintained under Earendil Works, so the harness study is not a vendor comparison between the authors’ own product and an external competitor.
The paper’s scope is also narrow. Visible reasoning text is neither a complete nor necessarily faithful representation of internal model reasoning, making semantic annotations less direct than deterministic telemetry. Tasks touched at most 4 files, and baseline success was close to 100%, so the absence of a success gain is partly a ceiling effect and cannot establish the same result for larger work. The paper preserves that boundary explicitly: Multiple approaches may be useful on repository-scale or architectural work where alternatives require genuine implementation and evaluation.
The direction and magnitude of these mechanisms are best treated as testable hypotheses to re-verify against the prompts, harness, and real tasks under examination, rather than as settled results.
arXiv 2608.01347v3 · Sarel Weinberger and Amir Hozez (PointFive) · v1 2026-08-02 / v3 2026-08-06 · 26 pages · preprint, not peer reviewed. Every figure, prompt string, mechanism description, and self-reported limitation in this article comes from its main text and six appendices. Figures 1 and 2 are not rendered in the arXiv HTML edition; their data was read from PDF pages 4 and 5. This article redraws the right panel of Figure 1 (the verification ladder) and Figure 2, preserving the per-level sample sizes and calls-per-run printed on the original axes; the left panel's per-level cost curve is a statistical detail this article does not cite.
Every row in Section 5 comes from direct access to the arXiv record, the repository URL given in the paper's references, the public repositories and commit history of the PointFiveLabs organization, and the TokenShift product page. The commands are reproducible. The checks state what was found and what it directly implies; they do not speculate about motive, and they distinguish strictly between "cannot be independently verified" and "is not true".
Created by Mario Zechner and now maintained under the Earendil Works organization. Exposes read, write, edit, and command execution as its four default tools. It has no affiliation with the authors' company, which is the basis for stating that the harness comparison is not a vendor self-comparison.
The argument that the orchestration layer rather than the model determines agent cost originates in Writer's arXiv 2607.06906. Section 3 here serves only as independent replication evidence and does not re-argue that case. The no-cache accounting question rests on treating caching as an architectural constraint rather than an afterthought, covered in this site's Prompt Caching material (no web edition). For the vendor-side result on system-prompt subtraction, see Section 2.8 of Getting the most out of the Claude Code harness, which is orthogonal to the user-side experiment reported here.