Jev vs OpenJEV
OpenJEV is the same idea as Jev with a chat LLM underneath: score each allowed answer by its log-probability. On a 0.5B backbone it shows what the mechanism alone buys.
undisclosed · undisclosed · Proprietary (hosted API)
Qwen2.5-0.5B-Instruct · 0.5B · Apache 2.0
Metatext decisions
Three hand-written workflows (support tickets, security incidents, agent traces), seven questions, about 123 judgments per model. Small, so treat a ±0.1 difference as a tie; every question has a majority baseline the model has to beat.
| Metric | Jev | OpenJEV |
|---|---|---|
| Decision accuracy | 0.863 | 0.427 |
| Questions over baseline | 6/7 | 3/7 |
| ECE raw → held-out fitted | 0.093 | 0.210 |
| p50 per case | 441 ms | 50 s |
| Extraction F1 | n/a | n/a |
Per-question accuracy
| Task | Question | Type | Baseline | Jev | OpenJEV |
|---|---|---|---|---|---|
| agent_traces | outcome | choice | 0.357 | 0.786 | 0.500 |
| agent_traces | needs_human | noul | 0.714 | 0.714 | 0.286 |
| customer_service | department | choice | 0.400 | 0.950 | 0.600 |
| customer_service | refund_requested | noul | 0.700 | 0.900 | 0.550 |
| customer_service | frustration | score | 0.500 | 0.950 | 0.300 |
| security_incidents | severity | choice | 0.312 | 0.875 | 0.500 |
| security_incidents | actively_exploited | noul | 0.688 | 0.938 | 0.312 |
typed-decisions
LAYA's public benchmark: four workflows × 100 JSON states × five typed questions = 2,000 decisions. Synthetic; gold is a teacher distribution and the label is its argmax. Large enough to carry an ordering.
| Metric | Jev | OpenJEV |
|---|---|---|
| Decision accuracy | 0.726 | — |
| Questions over baseline | 18/20 | — |
| ECE raw → held-out fitted | 0.079 → 0.049 | — |
| p50 per case | 405 ms | — |
Verdict
Jev, by 0.44 on the Metatext suite. OpenJEV at 0.427 is below every purpose-built encoder and takes about 50 seconds per case on CPU. The row is a scale substitution, not a like-for-like test, but it is enough to say the mechanism is not the advantage.
Related
The open reproductions of Jev (OpenJEV, LAYA, Von, and the Kotoba, Decider, Kev, Gavel checkpoints) explained by mechanism and measured on the same suite as the real thing.
Jev against LAYA on two suites: accuracy, calibration, latency, and what LAYA's in-domain fine-tune changes.