Jev calibration
A System One model is only useful if its probabilities mean something: p = 0.9 should be right nine times in ten. TypeSafe publishes no ECE. Third-party numbers range from 0.037 to 0.246 depending on the suite. Ours: 0.079 raw on typed-decisions, the best of any model before fitting, and 0.093 / 0.080 on the two Metatext runs.
| # | Model | Decisions | Beats baseline | ECE raw → held-out fitted | p50 / case |
|---|---|---|---|---|---|
| 1 | JevTypeSafehosted | 0.726 | 18/20 | 0.079→ 0.049 | 405 ms† |
| 2 | LAYA typed-decisionsConvai Innovations fine-tuned on this suite | 0.714→ 0.762 | 18/20 | 0.161→ 0.016 | 3.5 s† |
| 3 | LAYAConvai Innovations | 0.343 | 2/20 | 0.196→ 0.020 | 3.2 s† |
| 4 | Vonwfzyx | 0.461 | 5/20 | 0.249→ 0.027 | 597 ms |
| 5 | GLiClass multilang ultraKnowledgator correct payload, no prompt | 0.395 | 3/20 | 0.281→ 0.031 | 672 ms |
| 6 | GLiNER2 largeFastino | 0.486 | 9/20 | 0.300→ 0.029 | 3.8 s† |
| 7 | GLiClass large v3.0Knowledgator correct payload, no prompt | 0.446 | 7/20 | 0.315→ 0.093 | 977 ms |
| 8 | LAYA multilingualConvai Innovations | 0.345 | 2/20 | 0.322→ 0.073 | 1.2 s† |
| 9 | GLiNER 2.5 baseFastino | 0.398 | 3/20 | 0.415→ 0.059 | 1.3 s† |
Ranked by ece (lower is better). typed-decisions: LAYA's public benchmark: four workflows × 100 JSON states × five typed questions = 2,000 decisions. Synthetic; gold is a teacher distribution and the label is its argmax. Large enough to carry an ordering. Every model is zero-shot except LAYA typed-decisions, which was fine-tuned on this suite. Decisions is the task-weighted mean accuracy of the best configuration per model; ECE is the mean expected calibration error over tasks, and the arrow shows the held-out ECE after one temperature per primitive fitted on the other half. Latency is p50 per three-question customer-service case on four CPU cores (Jev: API round trip); † means taken from the benchmark run rather than a clean timing pass. Updated September 2026.
Same ranking on the other suite: Metatext decisions.
Frequently asked questions
Is Jev well calibrated?
On our suites, yes, and better than every open model untuned (ECE 0.079 on 2,000 decisions). Independent suites disagree with each other: 0.037 on a 102-item task (Ben Greenberg), 0.121 on 150 passages (WotAI), 0.246 on Banking77 with permuted options (nibzard). Calibration is suite-dependent; measure it on your own traffic.
Can the open models be calibrated?
Yes, cheaply. One temperature per primitive, fitted on half the data and judged on the other half, takes GLiNER2 large from 0.300 to 0.029 and LAYA typed-decisions from 0.161 to 0.016. It does not move accuracy except for LAYA typed-decisions, which ships under-confident and gains 0.05.
Related
An independent benchmark of TypeSafe's Jev against every open System One alternative: decision accuracy, calibration (ECE) and latency on two suites, 2,123 judgments, same harness for every row.
System One models ranked by expected calibration error on the Metatext suite, untuned, with the held-out ECE after temperature scaling where it was fitted.
Every published Jev benchmark figure in one place: TypeSafe's workflow evals and cookbooks, twenty independent evaluations, and the model cards of the open alternatives, each attributed to its URL and labelled by who measured it.