Stop premature “done”: decide continue / retry / change approach / ask / done after every step.
Agents declare victory early. The fix in the community is a supervisor: after every step, a fast typed decision — continue, retry, change_approach, ask_user, done — biased toward continue unless done is a confident yes. The supervisor is cheap; the loop it prevents restarting is not.
In development — the benchmark numbers on this page are measured; the Decide-1 model is in training.
Self-reported completion
The worker that did the work grades the work.
Retries without a reason
Nothing distinguishes “try again” from “try something else”.
Nobody asks the human
Ask_user is the option the loop never picks on its own.
- 1
One choice after each step
Over the five actions, with the diff, the test output and the original ask as state.
- 2
Bias the decision
Accept done only above 0.8; treat everything else as continue or retry.
- 3
Add nouls for the checks that matter
tests passed? scope respected? human input needed?
r = client.system_one(state={"ask": task, "diff": diff, "tests": test_output}, questions={
"next": {"type": "choice", "instructions": "What should the loop do now?",
"criteria": {"continue": "...", "retry": "...", "change_approach": "...", "ask_user": "...", "done": "..."}},
"tests_pass": {"type": "noul", "instructions": "The test output shows all tests passing."},
})
action = "done" if r["next"].probabilities["done"] > 0.8 else r["next"].choice- Choice — pick one option, a probability on each
- Noul — a calibrated yes/no probability
Pattern documented by jev-mcp and the Foreman project (a fast supervisor over Codex workers). On our agent-trace suite the completion question is the hardest in the benchmark for every model, Jev included — which is why it needs tuning on your traces.
Numbers about Jev are its authors' or the cited source's; numbers from the benchmark are ours, on one harness, with the raw JSON published. Full table.
Get early access — and tell us which decision you'd hand over first
We're building Metatext Decide with the first teams that use it. Leave your email, answer as much as you like, and say yes to a short interview if you want a say in what ships.
- Early access to Decide as it opens up
- A 20-minute interview if you want one — research, not a pitch
- The benchmark harness and raw results, today
- No spam — early-access updates only
Questions
- Why not let the worker decide?
- Because the failure mode is the worker being confidently wrong about its own output. A separate, calibrated judge with a bias to continue is the point.