Jev vs GLiNER2
GLiNER2 is a schema-conditioned encoder, not a System One model, but its classification schema maps one to one onto the Jev payload and it wins every open zero-shot row.
undisclosed · undisclosed · Proprietary (hosted API)
DeBERTa-v3 (span grid) · 340M · Apache 2.0
Metatext decisions
Three hand-written workflows (support tickets, security incidents, agent traces), seven questions, about 123 judgments per model. Small, so treat a ±0.1 difference as a tie; every question has a majority baseline the model has to beat.
| Metric | Jev | GLiNER2 large |
|---|---|---|
| Decision accuracy | 0.863 | 0.716 |
| Questions over baseline | 6/7 | 6/7 |
| ECE raw → held-out fitted | 0.093 | 0.261 |
| p50 per case | 441 ms | 627 ms |
| Extraction F1 | n/a | 0.968 |
Per-question accuracy
| Task | Question | Type | Baseline | Jev | GLiNER2 large |
|---|---|---|---|---|---|
| agent_traces | outcome | choice | 0.357 | 0.786 | 0.571 |
| agent_traces | needs_human | noul | 0.714 | 0.714 | 0.643 |
| customer_service | department | choice | 0.400 | 0.950 | 0.850 |
| customer_service | refund_requested | noul | 0.700 | 0.900 | 0.950 |
| customer_service | frustration | score | 0.500 | 0.950 | 0.950 |
| security_incidents | severity | choice | 0.312 | 0.875 | 0.438 |
| security_incidents | actively_exploited | noul | 0.688 | 0.938 | 0.812 |
typed-decisions
LAYA's public benchmark: four workflows × 100 JSON states × five typed questions = 2,000 decisions. Synthetic; gold is a teacher distribution and the label is its argmax. Large enough to carry an ordering.
| Metric | Jev | GLiNER2 large |
|---|---|---|
| Decision accuracy | 0.726 | 0.486 |
| Questions over baseline | 18/20 | 9/20 |
| ECE raw → held-out fitted | 0.079 → 0.049 | 0.300 → 0.029 |
| p50 per case | 405 ms | 3.8 s |
Per-question accuracy
| Task | Question | Type | Baseline | Jev | GLiNER2 large |
|---|---|---|---|---|---|
| td_agent_trace_observability | action | choice | 0.360 | 0.620 | 0.310 |
| td_agent_trace_observability | needs_review | noul | 0.550 | 0.600 | 0.450 |
| td_agent_trace_observability | outcome | choice | 0.500 | 0.800 | 0.250 |
| td_agent_trace_observability | risk | score | 0.340 | 0.630 | 0.460 |
| td_agent_trace_observability | urgency | score | 0.370 | 0.360 | 0.300 |
| td_customer_service | action | choice | 0.520 | 0.670 | 0.470 |
| td_customer_service | category | choice | 0.380 | 0.930 | 0.740 |
| td_customer_service | churn_risk | score | 0.470 | 0.860 | 0.660 |
| td_customer_service | needs_human | noul | 0.670 | 0.650 | 0.700 |
| td_customer_service | urgency | score | 0.420 | 0.840 | 0.610 |
| td_invoice_processing | discrepancy_severity | score | 0.470 | 0.570 | 0.160 |
| td_invoice_processing | disposition | choice | 0.450 | 0.650 | 0.450 |
| td_invoice_processing | duplicate | noul | 0.890 | 0.920 | 0.110 |
| td_invoice_processing | matches_order | noul | 0.540 | 0.960 | 0.620 |
| td_invoice_processing | urgency | score | 0.510 | 0.680 | 0.320 |
| td_security_incidents | credential_compromise | noul | 0.550 | 0.820 | 0.580 |
| td_security_incidents | disposition | choice | 0.740 | 0.790 | 0.740 |
| td_security_incidents | severity | score | 0.490 | 0.790 | 0.510 |
| td_security_incidents | true_positive | noul | 0.730 | 0.810 | 0.730 |
| td_security_incidents | urgency | score | 0.500 | 0.570 | 0.540 |
Verdict
Jev on decisions (0.726 vs 0.486 on typed-decisions; 0.863 vs 0.716 on the Metatext suite) and on untuned calibration. GLiNER2 on extraction (0.968 F1, which Jev cannot do at all), on price (Apache 2.0, your hardware) and, after one fitted temperature, on calibration (ECE 0.029 held-out).
Frequently asked questions
Can GLiNER2 replace Jev?
For extraction, Jev was never an option. For decisions, GLiNER2 large gives up about 0.15 to 0.24 of accuracy zero-shot; whether that is acceptable depends on whether your questions are near the majority baseline anyway.
Related
Micro-F1 on an eight-case entity extraction task for the models that have a span head. Jev, LAYA, Von and GLiClass never return spans; only the GLiNER family does.
Fastino's GLiNER2 large against Knowledgator's GLiClass large v3.0 on decisions, calibration, latency and extraction.
Every open-weights alternative to TypeSafe's Jev, ranked by measured decision accuracy on the 2,000-decision typed-decisions suite: LAYA, Von, GLiNER2, GLiClass, OpenJEV, plus the Jev-shaped checkpoints not yet run.