Evaluate agent workflows

The evaluation utilities test whether an agent selects and preserves the expected deterministic DICOM tools before answer synthesis.

They evaluate a tool-use workflow; they do not make a model’s free-form answer normative.

Build a database

The known fixture example uses:

uv run dicom-kb build-fixture \
  --edition 2026b \
  --db /tmp/dicom-kb-fixture.sqlite \
  --force

Run a reference case

uv run dicom-kb eval run \
  --agent reference \
  --case agent.ct.required_modules \
  --edition 2026b \
  --db /tmp/dicom-kb-fixture.sqlite \
  --out /tmp/dicom-kb-reference-run.json

reference is the default agent mode. Omit --case to run every committed case compatible with the selected database.

The output file contains compact agent-run transcript data suitable for the scorer.

Score the transcript

uv run dicom-kb eval score \
  /tmp/dicom-kb-reference-run.json

Write the scorecard to a file:

uv run dicom-kb eval score \
  /tmp/dicom-kb-reference-run.json \
  --output /tmp/dicom-kb-scorecard.json

By default, scoring exits nonzero when any run has issues. Generate a report without failing the command only when investigation requires it:

uv run dicom-kb eval score \
  /tmp/dicom-kb-reference-run.json \
  --no-fail-on-issues

The scorer accepts one AgentRun, a list of runs, or an object with a top-level runs list.

Run an external agent command

An external command receives JSON on standard input and must emit one AgentRun JSON object on standard output:

uv run dicom-kb eval run \
  --agent external \
  --external-command '/absolute/path/to/agent-adapter' \
  --external-provider provider-label \
  --external-model model-label \
  --external-timeout 300 \
  --case agent.ct.required_modules \
  --edition 2026b \
  --db /tmp/dicom-kb-fixture.sqlite \
  --out /tmp/dicom-kb-external-run.json

Provider and model labels are optional metadata. The timeout is in seconds and defaults to 300.

Treat the external command as executable code with access to the local environment. Use a reviewed adapter and avoid passing secrets that the case does not need.

Interpret failures

A failed case can indicate:

• the wrong tool was selected;
• a required tool call was omitted;
• arguments differed from the expected deterministic route;
• the transcript dropped references or other required evidence;
• the selected local database lacked the case’s required facts;
• the external adapter emitted invalid output.

Inspect the case id, expected tool trace, actual calls, database edition, and score details before changing a prompt or parser.

Use the suite as a release gate

The source release checklist expects:

• at least 100 prompt cases;
• deterministic expected tool traces;
• fixture coverage for offline regression;
• official-edition runs for release readiness;
• separate checks for structured tools and cited text fallback.

Keep agent evaluation separate from parser unit tests and official-artifact integration tests. Each catches a different failure mode.