Getting Started v1

1) Install

pnpm add -D @icodenet/eval-dashboards
# or
npm install -D @icodenet/eval-dashboards

2) Scaffold

# default profile: setup=all, runner=node, ci=github
eval-dashboards init --preset=agent-quality --write --playbook

# preview only
eval-dashboards init --preset=agent-quality --dry-run

Writes starter config, dataset, rubric, template artifact, CI snippet, and optional playbook with local-agent prompts.

3) Runner-specific flow

eval-dashboards init --preset=agent-quality --runner=vitest --write
pnpm vitest run
eval-dashboards lint --input=.evals_output
eval-dashboards check --input=.evals_output --min-pass-rate=0.9 --max-new-failures=0 --zero-critical
eval-dashboards report --input=.evals_output --reporter=html --reporter=json-summary --report-dir=eval-dashboard

eval-dashboards init --preset=agent-quality --runner=jest --write
pnpm jest
eval-dashboards lint --input=.evals_output
eval-dashboards check --input=.evals_output --min-pass-rate=0.9 --max-new-failures=0 --zero-critical
eval-dashboards report --input=.evals_output --reporter=html --reporter=json-summary --report-dir=eval-dashboard

eval-dashboards init --preset=agent-quality --runner=node --write
pnpm eval -- --offline --write-results
pnpm eval:emit-artifact
eval-dashboards lint --input=.evals_output
eval-dashboards check --input=.evals_output --min-pass-rate=0.9 --max-new-failures=0 --zero-critical
eval-dashboards report --input=.evals_output --reporter=html --reporter=json-summary --report-dir=eval-dashboard

eval-dashboards init --preset=agent-quality --runner=python --write
python -m pytest
eval-dashboards lint --input=.evals_output
eval-dashboards check --input=.evals_output --min-pass-rate=0.9 --max-new-failures=0 --zero-critical
eval-dashboards report --input=.evals_output --reporter=html --reporter=json-summary --report-dir=eval-dashboard

4) Local-agent prompt templates

Prompt A (dataset):
Update eval/datasets/agent-quality-cases.jsonl with stable ids and lifecycle values.
Add one positive + one adversarial case per changed feature.

Prompt B (rubric):
Edit eval/rubrics/agent-quality-rubrics.json.
If scoring criteria changes, bump rubricVersion and explain why.

Prompt C (judge calibration):
Run labelled sample and record judgeModel, rubricVersion, disagreement rate,
and concrete disagreement examples.

Prompt D (multiturn):
Add multiturn-trajectory rows with turns + toolCalls evidence.
Keep row ids stable for baseline comparisons.