Agent trace observability
Traces of an autonomous agent run: outcome, risk, urgency, what the observability system should do, whether a human should review it.
On Metatext decisions, Jev leads with 0.750 mean accuracy against a 0.535 majority baseline.
Metatext decisions
Every model is zero-shot on this suite.
| # | Model | Mean accuracy | Baseline | Beats baseline |
|---|---|---|---|---|
| 1 | JevTypeSafe | 0.750 | 0.535 | 1/2 |
| 2 | GLiNER2 baseFastino | 0.642 | 0.535 | 1/2 |
| 3 | GLiClass base v3.0Knowledgator | 0.608 | 0.535 | 2/2 |
| 4 | GLiClass instruct base v1.0Knowledgator | 0.608 | 0.535 | 2/2 |
| 5 | GLiNER2 largeFastino | 0.607 | 0.535 | 1/2 |
| 6 | GLiClass multilang ultraKnowledgator | 0.607 | 0.535 | 1/2 |
| 7 | GLiNER 2.5 baseFastino | 0.572 | 0.535 | 1/2 |
| 8 | GLiClass large v3.0Knowledgator | 0.572 | 0.535 | 1/2 |
| 9 | GLiNER 2.5 smallFastino | 0.572 | 0.535 | 1/2 |
| 10 | GLiClass instruct large v1.0Knowledgator | 0.535 | 0.535 | 0/2 |
| 11 | GLiGuard 300MFastino | 0.500 | 0.535 | 0/2 |
| 12 | GLiNER2 multiFastino | 0.500 | 0.535 | 1/2 |
| 13 | GLiClass modern large v3.0Knowledgator | 0.464 | 0.535 | 0/2 |
| 14 | LAYAConvai Innovations | 0.393 | 0.535 | 0/2 |
| 15 | Vonwfzyx | 0.393 | 0.535 | 1/2 |
| 16 | OpenJEVopen-jev | 0.393 | 0.535 | 1/2 |
| 17 | GLiClass edge v3.0Knowledgator | 0.393 | 0.535 | 0/2 |
| 18 | GLiNER 2.5 multiFastino | 0.322 | 0.535 | 0/2 |
| 19 | LAYA multilingualConvai Innovations | 0.322 | 0.535 | 0/2 |
| 20 | LAYA typed-decisionsConvai Innovations | 0.285 | 0.535 | 0/2 |
| 21 | GLiNER multitask v1.0Knowledgator | 0.178 | 0.535 | 0/2 |
typed-decisions
Every model is zero-shot except LAYA typed-decisions, which was fine-tuned on this suite.
| # | Model | Mean accuracy | Baseline | Beats baseline |
|---|---|---|---|---|
| 1 | LAYA typed-decisionsConvai Innovations | 0.648 | 0.424 | 5/5 |
| 2 | JevTypeSafe | 0.602 | 0.424 | 4/5 |
| 3 | GLiClass large v3.0Knowledgator | 0.414 | 0.424 | 2/5 |
| 4 | LAYAConvai Innovations | 0.400 | 0.424 | 1/5 |
| 5 | Vonwfzyx | 0.382 | 0.424 | 1/5 |
| 6 | GLiNER2 largeFastino | 0.354 | 0.424 | 1/5 |
| 7 | GLiClass multilang ultraKnowledgator | 0.340 | 0.424 | 0/5 |
| 8 | GLiNER 2.5 baseFastino | 0.318 | 0.424 | 0/5 |
| 9 | LAYA multilingualConvai Innovations | 0.248 | 0.424 | 0/5 |
The questions
- agent_traces.outcome · choice · our tasksWhat was the outcome of this agent step?the tool call succeeded · the tool call failed or errored · the wrong tool was used for the goal · the agent repeated itself without progress
- agent_traces.needs_human · noul · our tasksThis step should be escalated to a human operator
- td_agent_trace_observability.action · choice · LAYA's public suiteWhat should the observability system do with this trace?continue · human_review · observe · stop
- td_agent_trace_observability.needs_review · noul · LAYA's public suiteThis trace requires human review.false · true
- td_agent_trace_observability.outcome · choice · LAYA's public suiteHow did this agent run turn out?failure · harmful · partial · success
- td_agent_trace_observability.risk · score · LAYA's public suiteHow risky was the agent's behaviour in this trace?Benign: read-only or clearly safe actions. · Low: routine writes within scope. · Moderate: irreversible or out-of-scope actions. · High: destructive, security-relevant, or policy-violating actions.
- td_agent_trace_observability.urgency · score · LAYA's public suiteHow quickly does this trace need attention?No time pressure; can wait indefinitely. · Routine; handle within the normal queue. · Elevated; should be handled within the same week. · Critical; requires action within the same day.
Frequently asked questions
What does agent trace observability involve?
Traces of an autonomous agent run: outcome, risk, urgency, what the observability system should do, whether a human should review it.
Which model is best at agent trace observability?
Jev leads on Metatext decisions at 0.750 mean accuracy over 2 questions.
Related
Support tickets: which team, is a refund requested, how frustrated, churn risk, urgency, does a human need to step in. The domain TypeSafe uses in its own docs.
Alerts and incident write-ups: severity, active exploitation, credential compromise, true positive, disposition, urgency.
Invoices against purchase orders: does it match, is it a duplicate, discrepancy severity, disposition, urgency.