Extraction benchmark
Extraction is the fourth primitive and the one Jev does not have. Five labels (person, organization, location, date, money), eight cases, micro-F1. Not applicable is not zero: a model with no span head cannot be scored here.
| # | Model | Decisions | Beats baseline | ECE raw → held-out fitted | p50 / case | Extraction F1 |
|---|---|---|---|---|---|---|
| 1 | GLiNER2 multiFastino | 0.464 | 2/7 | 0.354 | 202 ms† | 1.000 |
| 2 | GLiNER2 largeFastino | 0.716 | 6/7 | 0.261 | 627 ms† | 0.968 |
| 3 | GLiNER 2.5 baseFastino | 0.643 | 5/7 | 0.251 | 332 ms† | 0.968 |
| 4 | GLiNER2 baseFastino | 0.631 | 3/7 | 0.350 | 190 ms† | 0.935 |
| 5 | GLiNER 2.5 multiFastino | 0.494 | 4/7 | 0.244 | 346 ms† | 0.885 |
| 6 | GLiNER 2.5 smallFastino | 0.574 | 4/7 | 0.172 | 132 ms† | 0.812 |
| 7 | GLiGuard 300MFastino | 0.537 | 3/7 | 0.230 | 155 ms† | 0.000 |
Ranked by extraction f1. Metatext decisions: Three hand-written workflows (support tickets, security incidents, agent traces), seven questions, about 123 judgments per model. Small, so treat a ±0.1 difference as a tie; every question has a majority baseline the model has to beat. Every model is zero-shot on this suite. Decisions is the task-weighted mean accuracy of the best configuration per model; ECE is the mean expected calibration error over tasks, and the arrow shows the held-out ECE after one temperature per primitive fitted on the other half. Latency is p50 per three-question customer-service case on four CPU cores (Jev: API round trip); † means taken from the benchmark run rather than a clean timing pass. Updated September 2026.
Same ranking on the other suite: typed-decisions.
Frequently asked questions
Can Jev extract entities?
No. Jev answers questions about text and never returns spans. If you need both decisions and extraction from one model, GLiNER2 is the only family measured here that does both.
Related
One table, every System One model: Jev, LAYA (three checkpoints), Von, OpenJEV, seven GLiNER2 and seven GLiClass checkpoints. Decision accuracy, calibration and latency on the Metatext suite, extraction F1 where the model has a span head.
Jev against GLiNER2 large: the best zero-shot open model on decisions, and the only one that also extracts spans.