assistant-ui reference integration v1

End-to-end case study: real harness, real artifact, reproducible commands.

Source material

Repro steps

  1. Run the assistant-ui eval harness with explicit judge/agent models.
  2. Emit eval-report/v1 using the adapter in the case study.
  3. Run eval-dashboards lint/check/report against that artifact.
JUDGE_MODEL=claude-sonnet-5 AGENT_MODEL=claude-sonnet-5 TRIALS=1 node src/cli.ts

eval-dashboards lint --input=.evals_output
eval-dashboards check --input=.evals_output --allow-blocked-baseline
eval-dashboards report --input=.evals_output --reporter=html --report-dir=eval-report
eval-dashboards publish --target=github-pages --dry-run --repo=owner/repo --report-dir=eval-report

What this validates

  • Runner-agnostic adapter path into eval-report/v1.
  • Real lint/check/report outputs on external-repo evidence.
  • Roadmap feedback grounded in observed integration friction.

Key lessons from this integration

  • Map local runner rows into eval-report/v1 with stable row ids first; add taxonomy completeness incrementally.
  • Use lint output as actionable backlog, not noise suppression. Missing evidence warnings usually indicate real harness gaps.
  • Keep publish flow reproducible in CI from day one, even if you start with dry-run mode.