Architecture · LikeC4

codeprobe

Mines tasks from a repo, replays agents against them in isolation, scores against oracles, and ranks configs

Open the interactive explorer → View on GitHub

built evolving planned research risk

The whole picture

System landscape

The system in context, then opened up into its containers.

codeprobe — system landscape — codeprobe — system landscape
index

codeprobe — containers (the assess → mine → run → score → interpret spine) — codeprobe — containers (the assess → mine → run → score → interpret spine)
codeprobeSystem

Mines tasks from a repo, replays agents against them in isolation, scores against oracles, and ranks configs

Risks & open questions — Risks & open questions
risks

Research & opt-in work — Research & opt-in work
research

Inside each box

Containers & components

Each part decomposed into the components that implement it. Every box links to its source in the interactive explorer.

CLI — command group & subcommands — CLI — command group & subcommands
cliContainer

cli/ — Click command group: assess, mine, run, interpret, experiment, probe, snapshot, …

Mining — repo history → tasks + consensus oracle — Mining — repo history → tasks + consensus oracle
miningContainer

mining/ — extract eval tasks from git history; build oracle ground truth via multi-backend consensus

Execution engine — isolated agent replay — Execution engine — isolated agent replay
executionContainer

core/ — drive each agent config against each task in an isolated worktree; collect telemetry

Agent adapters — the Adapter + Collector boundary — Agent adapters — the Adapter + Collector boundary
adaptersContainer

adapters/ — Adapter + Collector hybrid: drive each agent CLI, collect token/cost telemetry

Scoring / verifier — reward families & honesty guards — Scoring / verifier — reward families & honesty guards
scoringContainer

core/scoring/ — score agent output against oracle ground truth; honesty-linted reward families

Analysis — deterministic ranking + statistics + report — Analysis — deterministic ranking + statistics + report
analysisContainer

analysis/ — compare configs statistically and emit a ranked, formatted report

Calibration + QA — validity gates on the corpus — Calibration + QA — validity gates on the corpus
calibrationContainer

calibration/ + qa/ — validity gates: inter-curator agreement and cross-task leakage

Snapshot + trace — redacted shareable artifacts — Snapshot + trace — redacted shareable artifacts
snapshotContainer

snapshot/ + trace/ — deterministic redacted experiment snapshots; SQLite agent-trace recorder

Supporting libraries — domain, config, LLM, preambles — Supporting libraries — domain, config, LLM, preambles
supportContainer

models/ config/ loaders/ llm/ preambles/ net/ ratings/ — shared domain + IO plumbing

How it runs

Walkthrough flows

Dynamic views — the narrative spine of the system, step by step.

Walkthrough — mining tasks with a consensus oracle — Walkthrough — mining tasks with a consensus oracle
mineFlow

Multi-source consensus is on by default: every backend runs and a task ships only when ≥2 agree above the F1 threshold. This is the structural defense against the MCP-bias tautology — do not pass --no-consensus for MCP-vs-no-MCP comparisons.

Walkthrough — replaying an agent against a task — Walkthrough — replaying an agent against a task
runFlow

Each task runs in its own worktree pinned to the task commit; hide/scaffold isolation can stash local source so the agent must use MCP. Every adapter extracts token/cost telemetry — that is a contract, not best-effort.

Walkthrough — scoring output against the oracle — Walkthrough — scoring output against the oracle
scoreFlow

Reward families are honesty-linted: an F1-family branch must never silently fall back to recall, and ScoreResult must declare scorer_family=. Failures score as "incorrect" — they are never dropped.

Walkthrough — interpret with bias guards — Walkthrough — interpret with bias guards
interpretFlow

Before ranking, three bias detectors fire (backend_overlap / overshipping / no_independent_baseline). When no independent baseline exists the aggregate winner is suppressed by design — the table prints, the ranking does not.

Where it runs

Deployment

What runs where, and the process & data boundaries between the pieces.

Deployment — where each piece runs — Deployment — where each piece runs
deployment

Explore it live

The figures above are static exports. The interactive explorer lets you pan, zoom, follow relationships, and jump from any box to the source.

Open the interactive explorer →