Jev benchmark
This is the only public benchmark that runs the hosted Jev and the open alternatives (LAYA, Von, OpenJEV, GLiNER2, GLiClass) through one harness on the same tasks. TypeSafe's own dashboard scores Jev against LLMs with LLM-derived labels; this page scores it against the models you would actually swap it for.
| # | Model | Decisions | Beats baseline | ECE raw → held-out fitted | p50 / case |
|---|---|---|---|---|---|
| 1 | JevTypeSafehosted | 0.726 | 18/20 | 0.079→ 0.049 | 405 ms† |
| 2 | LAYA typed-decisionsConvai Innovations fine-tuned on this suite | 0.714→ 0.762 | 18/20 | 0.161→ 0.016 | 3.5 s† |
| 3 | GLiNER2 largeFastino | 0.486 | 9/20 | 0.300→ 0.029 | 3.8 s† |
| 4 | Vonwfzyx | 0.461 | 5/20 | 0.249→ 0.027 | 597 ms |
| 5 | GLiClass large v3.0Knowledgator correct payload, no prompt | 0.446 | 7/20 | 0.315→ 0.093 | 977 ms |
| 6 | GLiNER 2.5 baseFastino | 0.398 | 3/20 | 0.415→ 0.059 | 1.3 s† |
| 7 | GLiClass multilang ultraKnowledgator correct payload, no prompt | 0.395 | 3/20 | 0.281→ 0.031 | 672 ms |
| 8 | LAYA multilingualConvai Innovations | 0.345 | 2/20 | 0.322→ 0.073 | 1.2 s† |
| 9 | LAYAConvai Innovations | 0.343 | 2/20 | 0.196→ 0.020 | 3.2 s† |
Ranked by decision accuracy. typed-decisions: LAYA's public benchmark: four workflows × 100 JSON states × five typed questions = 2,000 decisions. Synthetic; gold is a teacher distribution and the label is its argmax. Large enough to carry an ordering. Every model is zero-shot except LAYA typed-decisions, which was fine-tuned on this suite. Decisions is the task-weighted mean accuracy of the best configuration per model; ECE is the mean expected calibration error over tasks, and the arrow shows the held-out ECE after one temperature per primitive fitted on the other half. Latency is p50 per three-question customer-service case on four CPU cores (Jev: API round trip); † means taken from the benchmark run rather than a clean timing pass. Updated September 2026.
Same ranking on the other suite: Metatext decisions.
Jev-shaped models not yet run
Surveyed on Hugging Face; each takes the System One request. Their own published figures are on their model pages, attributed; none is comparable to the table above until it has a row.
- Decider 2B1.9B · Apache 2.0
A decoder-backbone System One reproduction with a proper-scoring-rule SFT stage and a calibration-aware RL stage; serves the /v1/systemone request shape.
- open-jev DeBERTa-v3-large435M · Apache 2.0
Single-pass encoder with a per-option head over state and all questions in one sequence; 512-token context with the state cut to 256 tokens.
- Gavel base150M · MIT
Von's mechanism trained deliberately for the decision task, small and multilingual, with a fitted temperature and the card's own negative results.
- Kev 0.6B0.6B · Apache 2.0
A reproduction of the architecture described in "Jev's Architecture Unmasked"; the only one that measured the real Jev on its own frozen dev sets and published a research log.
- Jev schema scorer435M · Apache 2.0
Scores the literal Jev question JSON as the second sequence, so criteria are read at inference.
- System One Qwen3.5 4B scorer4.2B · Non-commercial
Pointwise scorer trained on 46k System One questions across nine families; reports its calibration separately from its accuracy.
- Qwen-Jeff B21.7B · Apache 2.0
Pointwise scorer with a CLI and an "automate if p ≥ 0.9588" gate calibrated to at most 5% error.
- RLCD ModernBERT 151M151M · Apache 2.0
A GLiClass-mechanism model with an RLCD-style objective; small enough to run in the browser (ONNX + WebGPU demo).
Frequently asked questions
What is Jev?
Jev is TypeSafe's System One model: a hosted decision model that takes a state and typed questions (Choice, Score, Noul) and returns a calibrated probability per answer in one round trip, without generating text.
How was Jev benchmarked here?
Through its public API (jev-latest, which resolved to jev-1.13.0), with the same state and questions every open model received, scored by the same code. Decision accuracy is a task-weighted mean; ECE is measured on the returned probabilities; latency is the round trip from a four-core CPU VM.
Does any open model match Jev?
Zero-shot, no: the best open model on typed-decisions is GLiNER2 large at 0.486 against Jev's 0.726. Fine-tuned in-domain, yes: LAYA typed-decisions reaches 0.714, within 0.012 of Jev, and 0.762 after one fitted temperature per primitive.
Related
Every open-weights alternative to TypeSafe's Jev, ranked by measured decision accuracy on the 2,000-decision typed-decisions suite: LAYA, Von, GLiNER2, GLiClass, OpenJEV, plus the Jev-shaped checkpoints not yet run.
One table, every System One model: Jev, LAYA (three checkpoints), Von, OpenJEV, seven GLiNER2 and seven GLiClass checkpoints. Decision accuracy, calibration and latency on the Metatext suite, extraction F1 where the model has a span head.
Every published Jev benchmark figure in one place: TypeSafe's workflow evals and cookbooks, twenty independent evaluations, and the model cards of the open alternatives, each attributed to its URL and labelled by who measured it.
Expected calibration error of Jev and every open System One model, raw and after one fitted temperature per primitive judged on a held-out fold. TypeSafe publishes no calibration number; this page does.