Architecture · LikeC4

migration-evals

Tiered-oracle funnel that grades agent batch-change diffs and reports a per-tier funnel + contamination split + merge-survival anchor

Open the interactive explorer → View on GitHub

built evolving planned research risk

The whole picture

System landscape

The system in context, then opened up into its containers.

migration-evals — system landscape — migration-evals — system landscape
index

migration-evals — containers (built vs planned) — migration-evals — containers (built vs planned)
evalsSystem

Tiered-oracle funnel that grades agent batch-change diffs and reports a per-tier funnel + contamination split + merge-survival anchor

Risks & open questions — Risks & open questions
risks

Planned & research work — Planned & research work
planned

Inside each box

Containers & components

Each part decomposed into the components that implement it. Every box links to its source in the interactive explorer.

Tiered oracle funnel (M1) — components — Tiered oracle funnel (M1) — components
funnelContainer

Cascades each trial T0→T1→T2→T2b→T3→T4 cheapest-first, short-circuits on first failure, aggregates per-tier verdicts into a FunnelResult

Task definition & intake — components — Task definition & intake — components
intakeContainer

YAML run configs + recipe specs define the migration; changeset providers stage (repo, base_commit, patch.diff) for the funnel

Build-harness synthesis (M2) — components — Build-harness synthesis (M2) — components
harnessContainer

LLM-inferred Dockerfile + build/test recipe, content-hash cached so repeat runs cost zero adapter calls

Execution & inference adapters — components — Execution & inference adapters — components
adaptersContainer

Protocol-typed seam between the funnel and any runtime; cassette stand-ins keep the suite offline, live adapters hit Docker / LLM APIs

Synthetic task generation (M3 / M9) — components — Synthetic task generation (M3 / M9) — components
syntheticContainer

Procedural repos with exact AST ground truth at zero marginal cost; generator and oracle kept code-disjoint (D5)

Runner, stamping & validity gates — components — Runner, stamping & validity gates — components
runnerContainer

Drives the config-defined run loop, writes one stamped result.json per trial, classifies failures, and gates publication

Reporting & gold anchor — components — Reporting & gold anchor — components
reportingContainer

Aggregates stamped results into a funnel report + contamination split + merge-survival correlation

How it runs

Walkthrough flows

Dynamic views — the narrative spine of the system, step by step.

Walkthrough — one eval run, config to stamped result — Walkthrough — one eval run, config to stamped result
evalRun

The smoke path runs end-to-end from a fresh clone with no API keys, no Docker: the cassette adapters replay sandbox + judge so every tier is exercised offline. Stages are filtered by the config (compile, tests, judge for the smoke fixtures).

Walkthrough — the funnel cascade (live adapters) — Walkthrough — the funnel cascade (live adapters)
funnelCascade

Cheapest-first, short-circuit on first fail: ~40% die at T0/T1 for $0.01, only the residual ambiguous cases reach the $0.08 LLM judge. Each failing tier stamps exactly one failure_class so a 40-repo regression triages in under 2h.

Walkthrough — synthetic generation + AST ground truth — Walkthrough — synthetic generation + AST ground truth
syntheticFlow

Procedural repos give exact AST-level truth at zero marginal cost. The T2b oracle checks 5 primitives; the generator emits 10 — the ≤50% overlap (D5) keeps generator and oracle code-disjoint so a green oracle is not a tautology.

Walkthrough — report, contamination split, gold anchor — Walkthrough — report, contamination split, gold anchor
reportFlow

No headline number leaves the project until the publication gate is green: every result stamped, no cassette_miss, and the eval_broken gate (Phi<0.7 or CI-low<0.5) not tripped. The contamination split is mandatory on every report.

Where it runs

Deployment

What runs where, and the process & data boundaries between the pieces.

Deployment — where each piece runs — Deployment — where each piece runs
deployment

Explore it live

The figures above are static exports. The interactive explorer lets you pan, zoom, follow relationships, and jump from any box to the source.

Open the interactive explorer →