Benchmarks
Nine controlled comparisons, including the miss.
BuildBrake and direct Codex received identical tasks, models, and starting commits. Every result was checked by an independent test committed before either run.
New input35.6%lower
Total input32.8%lower
Commands52.0%lower
Proof rate9 / 9both approaches
What this proves
In this controlled sample, BuildBrake used fewer aggregate tokens and commands without reducing the proof rate. It does not prove that every task, repository, or model will save tokens.
Individual results
| Trial | Task | BuildBrake new | Direct new | Difference | Outcome |
|---|---|---|---|---|---|
| 1 | Exact CSS declaration | 6,706 | 11,953 | 43.9% lower | Proved / Proved |
| 2 | Exact copy replacement | 6,027 | 7,278 | 17.2% lower | Proved / Proved |
| 3 | Accessibility attribute | 5,993 | 7,403 | 19.0% lower | Proved / Proved |
| 4 | JavaScript behavior | 7,159 | 11,393 | 37.2% lower | Proved / Proved |
| 5 | Python logic | 7,378 | 8,446 | 12.6% lower | Proved / Proved |
| 6 | Two-file Python refactor | 7,989 | 25,106 | 68.2% lower | Proved / Proved |
| 7 | Implicit implementation file | 6,481 | 5,342 | 21.3% higher | Proved / Proved |
| 8 | Bookmark feature | 11,483 | 21,672 | 47.0% lower | Proved / Proved |
| 9 | Noisy repository backend | 12,668 | 13,031 | 2.8% lower | Proved / Proved |
Method
- Start two isolated worktrees from the same pre-change commit.
- Commit a failing acceptance check before either agent runs.
- Give both agents the exact same task, model, reasoning setting, and sandbox.
- Run one through BuildBrake and one directly through Codex.
- Execute the same independent proof after both runs.
- Compare new input, cached input, total input, commands, changed files, runtime, and outcome.
Why trial 7 matters
Trial 7 is the honest counterexample. BuildBrake used 21.3 percent more new input tokens than direct Codex, although it still used fewer total tokens and commands. The aggregate claim includes this loss rather than hiding it.
Scope and limitations
- Nine trials remain a small sample.
- Most trials used small dependency-free projects.
- Only one trial tested a repository with substantial tracked noise.
- The results support the context and verification strategy, not a universal savings guarantee.
The raw narrative and complete per-trial notes remain available in the source repository.