Jev vs Von
Von presents every option as a hypothesis and softmaxes the entailment logits. It is the cleanest test of whether Jev's advantage is its mechanism: an NLI head is the obvious way to build one.
undisclosed · undisclosed · Proprietary (hosted API)
ModernBERT-large NLI head · 395M · Apache 2.0
Metatext decisions
Three hand-written workflows (support tickets, security incidents, agent traces), seven questions, about 123 judgments per model. Small, so treat a ±0.1 difference as a tie; every question has a majority baseline the model has to beat.
| Metric | Jev | Von |
|---|---|---|
| Decision accuracy | 0.863 | 0.509 |
| Questions over baseline | 6/7 | 4/7 |
| ECE raw → held-out fitted | 0.093 | 0.234 → 0.122 |
| p50 per case | 441 ms | 597 ms |
| Extraction F1 | n/a | n/a |
Per-question accuracy
| Task | Question | Type | Baseline | Jev | Von |
|---|---|---|---|---|---|
| agent_traces | outcome | choice | 0.357 | 0.786 | 0.500 |
| agent_traces | needs_human | noul | 0.714 | 0.714 | 0.286 |
| customer_service | department | choice | 0.400 | 0.950 | 0.650 |
| customer_service | refund_requested | noul | 0.700 | 0.900 | 0.950 |
| customer_service | frustration | score | 0.500 | 0.950 | 0.300 |
| security_incidents | severity | choice | 0.312 | 0.875 | 0.562 |
| security_incidents | actively_exploited | noul | 0.688 | 0.938 | 0.438 |
typed-decisions
LAYA's public benchmark: four workflows × 100 JSON states × five typed questions = 2,000 decisions. Synthetic; gold is a teacher distribution and the label is its argmax. Large enough to carry an ordering.
| Metric | Jev | Von |
|---|---|---|
| Decision accuracy | 0.726 | 0.461 |
| Questions over baseline | 18/20 | 5/20 |
| ECE raw → held-out fitted | 0.079 → 0.049 | 0.249 → 0.027 |
| p50 per case | 405 ms | 597 ms |
Per-question accuracy
| Task | Question | Type | Baseline | Jev | Von |
|---|---|---|---|---|---|
| td_agent_trace_observability | action | choice | 0.360 | 0.620 | 0.320 |
| td_agent_trace_observability | needs_review | noul | 0.550 | 0.600 | 0.450 |
| td_agent_trace_observability | outcome | choice | 0.500 | 0.800 | 0.440 |
| td_agent_trace_observability | risk | score | 0.340 | 0.630 | 0.370 |
| td_agent_trace_observability | urgency | score | 0.370 | 0.360 | 0.330 |
| td_customer_service | action | choice | 0.520 | 0.670 | 0.260 |
| td_customer_service | category | choice | 0.380 | 0.930 | 0.720 |
| td_customer_service | churn_risk | score | 0.470 | 0.860 | 0.680 |
| td_customer_service | needs_human | noul | 0.670 | 0.650 | 0.670 |
| td_customer_service | urgency | score | 0.420 | 0.840 | 0.610 |
| td_invoice_processing | discrepancy_severity | score | 0.470 | 0.570 | 0.450 |
| td_invoice_processing | disposition | choice | 0.450 | 0.650 | 0.470 |
| td_invoice_processing | duplicate | noul | 0.890 | 0.920 | 0.110 |
| td_invoice_processing | matches_order | noul | 0.540 | 0.960 | 0.540 |
| td_invoice_processing | urgency | score | 0.510 | 0.680 | 0.490 |
| td_security_incidents | credential_compromise | noul | 0.550 | 0.820 | 0.550 |
| td_security_incidents | disposition | choice | 0.740 | 0.790 | 0.280 |
| td_security_incidents | severity | score | 0.490 | 0.790 | 0.420 |
| td_security_incidents | true_positive | noul | 0.730 | 0.810 | 0.730 |
| td_security_incidents | urgency | score | 0.500 | 0.570 | 0.340 |
Verdict
Jev by 0.35 on the Metatext suite and 0.27 on typed-decisions. Von lands level with the GLiClass row that uses the same NLI mapping, which says the borrowed NLI head is doing the work, and it fails Score, like every NLI-mapped row. Its README's comparison table is not produced by anything in its repository.
Frequently asked questions
Is Von's "91.23%" a Jev benchmark?
No. It is Von's held-out NLI mixture (ANLI, WANLI, MNLI, SNLI). On a shared decision suite (jabr v2) its own card has Jev at 96.6% and Von at 72.0%.
Related
The two open models that take the Jev request unchanged, on the same suites.
The open reproductions of Jev (OpenJEV, LAYA, Von, and the Kotoba, Decider, Kev, Gavel checkpoints) explained by mechanism and measured on the same suite as the real thing.
Every open-weights alternative to TypeSafe's Jev, ranked by measured decision accuracy on the 2,000-decision typed-decisions suite: LAYA, Von, GLiNER2, GLiClass, OpenJEV, plus the Jev-shaped checkpoints not yet run.