Model cardAlexWortega
AlexWortega/openjev
Measured on: MMLU, ARC, GPQA-diamond, MNLI
- openjev-4B zero-shot MMLU / ARC-C / GPQA-d
- 0.472 / 0.592 / 0.273
Author-reported; Jev values were read off a chart, so no numeric Jev row.
OpenJEV
open-jevOpen weights
The open-source implementation of Jev's mechanism: score each allowed answer by its log-probability under a chat LLM, then assemble the distribution. Measured here on a 0.5B backbone, so it is a scale substitution, not a like-for-like row.
- typed-decisions
- —
- Metatext suite
- 0.427
- ECE untuned
- 0.210
- p50 per case
- 50 s
not run
3/7 over baseline
four CPU cores
Specifications
- Checkpoint
- Qwen2.5-0.5B-Instruct (bf16) via open-jev
- Publisher
- open-jev
- Mechanism
- LLM log-probability scoring (Jev mechanism)
- Backbone
- Qwen2.5-0.5B-Instruct
- Parameters
- 0.5B
- License
- Apache 2.0
- Runs as
- open weights, self-hosted
- Library
- open-jev
- Primitives
- Choice, Score, Noul
- Pricing
- none — your compute
Metatext decisionsrank 19 of 21
Full leaderboard →Per-question accuracy — default payload, systemone mode, 120 judgments
| Task | Question | Type | Accuracy | Baseline | Beats? |
|---|---|---|---|---|---|
| agent_traces | outcome | choice | 0.500 | 0.357 | ✅ |
| agent_traces | needs_human | noul | 0.286 | 0.714 | ❌ |
| customer_service | department | choice | 0.600 | 0.400 | ✅ |
| customer_service | refund_requested | noul | 0.550 | 0.700 | ❌ |
| customer_service | frustration | score | 0.300 | 0.500 | ❌ |
| security_incidents | severity | choice | 0.500 | 0.312 | ✅ |
| security_incidents | actively_exploited | noul | 0.312 | 0.688 | ❌ |
Published numbers
Reported by the authors or by third parties, on their own suites. Not comparable to the leaderboard above; attributed so you can check them.
Frequently asked questions
How does OpenJEV compare to Jev?
On the Metatext suite OpenJEV scores 0.427 against Jev's 0.863.
Is OpenJEV open source?
Yes. Qwen2.5-0.5B-Instruct (bf16) via open-jev is released under Apache 2.0 and runs on your own hardware with the open-jev library.
Can OpenJEV extract entities?
No. OpenJEV answers questions about text and never returns spans; use a GLiNER-family model for extraction.