Architecture · LikeC4

CodeScaleBench

Authors tasks, launches agent runs across retrieval arms on Harbor, scores them with verifier + judge, and publishes auditable snapshots

Open the interactive explorer → View on GitHub

built evolving planned research risk

The whole picture

System landscape

The system in context, then opened up into its containers.

CodeScaleBench — system landscape — CodeScaleBench — system landscape
index

CodeScaleBench — containers & data (built vs research) — CodeScaleBench — containers & data (built vs research)
csbSystem

Authors tasks, launches agent runs across retrieval arms on Harbor, scores them with verifier + judge, and publishes auditable snapshots

Risks & open questions — Risks & open questions
risks

Research track — agent-observatory extraction — Research track — agent-observatory extraction
research

Inside each box

Containers & components

Each part decomposed into the components that implement it. Every box links to its source in the interactive explorer.

Task authoring & calibration — components — Task authoring & calibration — components
authoringContainer

scripts/authoring/ — turns raw repos + templates into validated, calibrated benchmark tasks

Run execution (csb CLI) — components — Run execution (csb CLI) — components
executionContainer

lib/ + scripts/running/ — resolves a run config, injects the retrieval arm, and drives Harbor across the matrix

Grading — verifier + judge stack — Grading — verifier + judge stack
gradingContainer

scripts/evaluation/ + csb_metrics/judge/ — the dual-verifier oracle and the multi-round LLM judge

Metrics extraction — components — Metrics extraction — components
metricsContainer

scripts/csb_metrics/ — the structured pipeline from raw trial artifacts to scored, comparable metrics

Analysis, reporting & publishing — components — Analysis, reporting & publishing — components
reportingContainer

scripts/analysis/ + scripts/publishing/ — turns scored metrics into comparisons, galleries, and frozen auditable snapshots

Reliability Observatory — components — Reliability Observatory — components
observatoryContainer

observatory/ — extracts behavioural signals from trajectories, annotates each trial against a 23-category taxonomy, trains a classifier, and reports reliability

Infra & maintenance gates — components — Infra & maintenance gates — components
infraContainer

scripts/infra/ + scripts/maintenance/ — pre-run readiness, capacity planning, and the repo-health pre-commit gate

How it runs

Walkthrough flows

Dynamic views — the narrative spine of the system, step by step.

Walkthrough — authoring & calibrating a task — Walkthrough — authoring & calibrating a task
authorTask

A task is only admitted once its artifact verifier passes the triad. The triad gates the UPSTREAM PUSH, not corpus existence — a failing task stays put until its fix bead lands.

Walkthrough — a benchmark run end-to-end — Walkthrough — a benchmark run end-to-end
benchmarkRun

One config expands to a matrix of arms (baseline / sourcegraph / augment / github). Always run check_infra first — OAuth credentials existing is not the same as accounts being usable.

Walkthrough — scoring & metrics extraction — Walkthrough — scoring & metrics extraction
scoreRun

Dual-score: the deterministic artifact verifier (A/B/C) is the spine; the LLM judge runs on a cross-provider backend to avoid same-model bias and contributes a weighted rubric score.

Walkthrough — publishing an auditable snapshot — Walkthrough — publishing an auditable snapshot
publishSnapshot

verify_snapshot is the gate: it checks symlink integrity + completeness before a public release. The frozen trees ship via GitHub Releases; the repo carries the manifest + browse.html + the compare gallery.

Where it runs

Deployment

What runs where, and the process & data boundaries between the pieces.

Deployment — where each piece runs — Deployment — where each piece runs
deployment

Explore it live

The figures above are static exports. The interactive explorer lets you pan, zoom, follow relationships, and jump from any box to the source.

Open the interactive explorer →