What’s the next bottleneck?

look counter
inspect C1
open_joint T1
run_tap both 1
briefinggolookinspectrun_tapwaitset_valveplace_bucketopen_jointclose_jointrepairfinish
run_tap cold 3→






Six faults

trap_clog
branch_clog
supply_leak
drain_leak
aerator_clog
valve_not_openSolvable. Not gameable.
- Reference plumber
- Services everything, never tests
- Five exploits
- Mean
Table
| Reference plumber | Services everything, never tests | Five exploits | |
|---|---|---|---|
| Level 1 | 0.96 (0.93–0.98) | 0.41 (0.00–0.85) | 0.00 (0.00–0.00) |
| Level 2 | 0.96 (0.93–0.98) | 0.37 (0.00–0.85) | 0.00 (0.00–0.00) |
| Level 3 | 0.95 (0.93–0.97) | 0.43 (0.00–0.85) | 0.00 (0.00–0.00) |
| Level 4 | 0.94 (0.91–0.98) | 0.51 (0.06–0.85) | 0.00 (0.00–0.00) |
Mean (lowest–highest) over 100 houses per level; the five exploits pooled.
Where models break
- Reference plumber
- Services everything, never tests
- Codex, default model
5 of 8level 3–4 jobs reported done with a fault still in the house
| Model | Levels | Jobs | Score | Every fault fixed | Faults named right | Tokens read per job | Written | Tool calls | Time |
|---|---|---|---|---|---|---|---|---|---|
| Codex, default model | 1–2 | 4 | 0.90 | 4 of 4 | 4 of 4 | 428k (92% cached) | 1.4k | 18 | 50 s |
| Codex, default model | 3–4 | 8 | 0.51 | 2 of 8 | 2 of 8 | 731k (94% cached) | 2.0k | 26 | 57 s |
Scored by replay
- AgentOpenCode + model
12 MCP tools↔picture + text- Housesimulator + renderer
action log→- Verifierreplays it from the seed
- →
- Score 0–1
repair ×0.700.150.100.05
- Base
- Dry floor
- Time left
- Right diagnosis
repair: share of faults fixed, −0.25 per part left worse
harbor run --repo valuhaka/plumberbench -d plumberbench -a opencode -m MODEL
Try it
Junction 2026 · AfterQuery, Break the Agent