Architecture · LikeC4
Agent Diagnostics
Turns raw agent-trial output into a labeled, queryable reliability dataset (11,995 trials, 40 categories / 11 dimensions)
The whole picture
System landscape
The system in context, then opened up into its containers.
Agent Diagnostics — system landscape
index
Agent Diagnostics — containers
observatorySystemTurns raw agent-trial output into a labeled, queryable reliability dataset (11,995 trials, 40 categories / 11 dimensions)
Risks & open questions
risks
Evolving contracts
evolvingInside each box
Containers & components
Each part decomposed into the components that implement it. Every box links to its source in the interactive explorer.
Core package (src/agent_diagnostics) — components
coreContainersrc/agent_diagnostics — the extraction / annotation / training / serving engine behind the CLI
LLM annotator — backends, prompt & cache
llmViewReads actual trajectories and classifies with Claude. Three interchangeable backends behind one dispatch interface; signals redacted before the prompt to prevent label leakage
Signal extraction — registry & cache
signalsViewReads each trial dir (result.json + trajectory.json) into a 31-field TrialSignals row; classifies tool calls via the tool registry
Golden-corpus & docs scripts
scriptsViewOut-of-CI tooling: build / curate the committed golden regression corpus, and regenerate the schema docs from the authoritative code
How it runs
Walkthrough flows
Dynamic views — the narrative spine of the system, step by step.
Walkthrough — ingest (trials → signals)
ingestFlowPoint the tool at a directory of trial dirs. The extractor reads result.json + trajectory.json per trial, classifies tool calls via the per-agent registry, and writes one 31-field row per trial; the content-hash cache skips trials whose bytes are unchanged.
Walkthrough — the label pipeline
labelFlowHeuristic rules label the whole corpus instantly; the LLM labels a sample from the real trajectories. blend merges them (heuristic-only share capped) into a training set; train cross-validates per category and stores held-out F1; ensemble trusts deterministic tier-1 rules plus tier-2 classifier categories that clear the F1 (and optional ECE) gate.
Walkthrough — LLM annotation of one trial
llmAnnotateFlowThe prompt builder strips outcome-bearing signal fields (reward / passed / exception) so the model cannot read the answer, then truncates the trajectory head+tail. The annotation cache is keyed on sha256(prompt + model id), so a re-run with the same prompt and model is free; errors are never cached and re-run.
Walkthrough — query & export the dataset
serveFlowA fresh clone can query immediately: the query engine prefers data/export/*.parquet over the JSONL working set. export rewrites Parquet deterministically (fixed created_by, native list columns) and writes a MANIFEST with schema version, row counts, sha256, and source commit.
Where it runs
Deployment
What runs where, and the process & data boundaries between the pieces.
Deployment — where each piece runs
deploymentExplore it live
The figures above are static exports. The interactive explorer lets you pan, zoom, follow relationships, and jump from any box to the source.