Schema / Taxonomy v1

Artifact-first contract for runner-agnostic evals.

Required minimum

{
  "schemaVersion": "eval-report/v1",
  "run": { "id": "local-run", "generatedAt": "2026-01-01T00:00:00.000Z" },
  "suites": [{ "id": "quality", "total": 1, "passed": 1, "failed": 0 }],
  "rows": [{ "id": "case-001", "suite": "quality", "passed": true }]
}

Taxonomy-complete row targets

  • Classification: kind, severity, category
  • Agent evidence: turns, toolCalls, agentVersion, promptVersion
  • Judge evidence: judgeModel, judgeVerdict, judgeReasoning, axisScores
  • Stable IDs: datasetId, scenarioId, rubricId
  • Trace links (optional): trace.traceId, trace.spanId, trace.traceUrl, trace.spanUrl

Suite governance targets

  • riskArea, datasetVersion, rubricVersion, graders
  • gate.mode and explicit thresholds
  • Use additive optional fields for v1 evolution.

Deep references: docs/artifact-format.md, docs/taxonomy.md, schemas/eval-report-v1.schema.json.