System One model benchmark
A System One model does not generate text. It reads a state, answers typed questions, and returns a probability for every allowed answer. That makes it cheap and fast, and it makes accuracy and calibration the only two numbers that matter. This benchmark measures both, for every model that speaks the shape.
| # | Model | Decisions | Beats baseline | ECE raw → held-out fitted | p50 / case |
|---|---|---|---|---|---|
| 1 | JevTypeSafehosted | 0.726 | 18/20 | 0.079→ 0.049 | 405 ms† |
| 2 | LAYA typed-decisionsConvai Innovations fine-tuned on this suite | 0.714→ 0.762 | 18/20 | 0.161→ 0.016 | 3.5 s† |
| 3 | GLiNER2 largeFastino | 0.486 | 9/20 | 0.300→ 0.029 | 3.8 s† |
| 4 | Vonwfzyx | 0.461 | 5/20 | 0.249→ 0.027 | 597 ms |
| 5 | GLiClass large v3.0Knowledgator correct payload, no prompt | 0.446 | 7/20 | 0.315→ 0.093 | 977 ms |
| 6 | GLiNER 2.5 baseFastino | 0.398 | 3/20 | 0.415→ 0.059 | 1.3 s† |
| 7 | GLiClass multilang ultraKnowledgator correct payload, no prompt | 0.395 | 3/20 | 0.281→ 0.031 | 672 ms |
| 8 | LAYA multilingualConvai Innovations | 0.345 | 2/20 | 0.322→ 0.073 | 1.2 s† |
| 9 | LAYAConvai Innovations | 0.343 | 2/20 | 0.196→ 0.020 | 3.2 s† |
Ranked by decision accuracy. typed-decisions: LAYA's public benchmark: four workflows × 100 JSON states × five typed questions = 2,000 decisions. Synthetic; gold is a teacher distribution and the label is its argmax. Large enough to carry an ordering. Every model is zero-shot except LAYA typed-decisions, which was fine-tuned on this suite. Decisions is the task-weighted mean accuracy of the best configuration per model; ECE is the mean expected calibration error over tasks, and the arrow shows the held-out ECE after one temperature per primitive fitted on the other half. Latency is p50 per three-question customer-service case on four CPU cores (Jev: API round trip); † means taken from the benchmark run rather than a clean timing pass. Updated September 2026.
Same ranking on the other suite: Metatext decisions.
Jev-shaped models not yet run
Surveyed on Hugging Face; each takes the System One request. Their own published figures are on their model pages, attributed; none is comparable to the table above until it has a row.
- Decider 2B1.9B · Apache 2.0
A decoder-backbone System One reproduction with a proper-scoring-rule SFT stage and a calibration-aware RL stage; serves the /v1/systemone request shape.
- open-jev DeBERTa-v3-large435M · Apache 2.0
Single-pass encoder with a per-option head over state and all questions in one sequence; 512-token context with the state cut to 256 tokens.
- Gavel base150M · MIT
Von's mechanism trained deliberately for the decision task, small and multilingual, with a fitted temperature and the card's own negative results.
- Kev 0.6B0.6B · Apache 2.0
A reproduction of the architecture described in "Jev's Architecture Unmasked"; the only one that measured the real Jev on its own frozen dev sets and published a research log.
- Jev schema scorer435M · Apache 2.0
Scores the literal Jev question JSON as the second sequence, so criteria are read at inference.
- System One Qwen3.5 4B scorer4.2B · Non-commercial
Pointwise scorer trained on 46k System One questions across nine families; reports its calibration separately from its accuracy.
- Qwen-Jeff B21.7B · Apache 2.0
Pointwise scorer with a CLI and an "automate if p ≥ 0.9588" gate calibrated to at most 5% error.
- RLCD ModernBERT 151M151M · Apache 2.0
A GLiClass-mechanism model with an RLCD-style objective; small enough to run in the browser (ONNX + WebGPU demo).
Frequently asked questions
What is a System One model?
A model that answers typed questions (Choice, Score, Noul) about a state with a probability distribution instead of generated text. TypeSafe coined the term for Jev; LAYA, Von, Decider, Kev and Gavel are open reproductions of the shape.
What does the benchmark measure?
Decision accuracy against a gold label, how many questions beat the majority baseline, expected calibration error on the returned probabilities (raw and after one fitted temperature per primitive, judged on a held-out fold), and latency per case.
Which suites are used?
LAYA's public typed-decisions suite (2,000 decisions across four workflows) carries the ordering; the smaller Metatext suite (123 hand-written judgments, three workflows plus an extraction task) is the sanity check and the only one that scores span extraction.
Related
One table, every System One model: Jev, LAYA (three checkpoints), Von, OpenJEV, seven GLiNER2 and seven GLiClass checkpoints. Decision accuracy, calibration and latency on the Metatext suite, extraction F1 where the model has a span head.
An independent benchmark of TypeSafe's Jev against every open System One alternative: decision accuracy, calibration (ECE) and latency on two suites, 2,123 judgments, same harness for every row.
Leaderboard for LAYA's public typed-decisions suite: Jev, LAYA, Von, GLiNER2 and GLiClass on 2,000 decisions across four workflows, with per-question detail and calibration.
Leaderboard for the Metatext suite: three hand-written workflows, seven questions, 123 judgments, plus extraction, for Jev and every GLiNER2, GLiClass, LAYA, Von and OpenJEV checkpoint.