CLI reference¶
Every command takes --home <dir> (or HARNESSLAB_HOME) to choose the data directory; the default is ./.harnesslab. Run harnesslab <command> --help for the same text in the terminal.
Harness Lab: run the same coding task against several agent harnesses and compare verified results.
Usage:
Options:
--home <path>: Harness Lab data directory (default ./.harnesslab). [env var: HARNESSLAB_HOME]--install-completion: Install completion for the current shell.--show-completion: Show completion for the current shell, to copy it or customize the installation.--help: Show this message and exit.
Commands:
version: Print the Harness Lab version.init: Scaffold a lab directory with the demo...doctor: Check python, git, codex, claude and the...run: Run every task of a suite against one or...serve: Start the local dashboard.suite: Inspect and sanity-check task suites.experiment: Inspect and export past experiments.sweep: Configuration sweeps: search model x...grow: Growing Harness: grow a harness bundle...harness: Inspect and validate harness bundles.ablate: Component ablation: test every part of a...
harnesslab version¶
Print the Harness Lab version.
Usage:
Options:
--help: Show this message and exit.
harnesslab init¶
Scaffold a lab directory with the demo suite, sweep templates and a pricing example.
Usage:
Arguments:
directory: Directory to scaffold (created if missing). [default: .]
Options:
--force: Overwrite existing files.--help: Show this message and exit.
harnesslab doctor¶
Check python, git, codex, claude and the database.
Usage:
Options:
--help: Show this message and exit.
harnesslab run¶
Run every task of a suite against one or more harness variants.
Usage:
Arguments:
target: A suite.yaml, an experiment.yaml, or a bundled suite name such as 'demo'. [required]
Options:
-v, --variants <str>: Comma-separated variant ids.-t, --tasks <str>: Comma-separated task ids (default: all).-r, --repetitions <int range>: [x>=1]-p, --parallelism <int range>: [x>=1]-n, --name <str>: Experiment name.--keep-worktrees: Do not delete worktrees after runs.--pricing <path>: pricing.yaml for cost estimates.--plugin <str>: Python module that registers custom runners (repeatable).--help: Show this message and exit.
harnesslab serve¶
Start the local dashboard.
Usage:
Options:
--host <str>: [default: 127.0.0.1]--port <int>: [default: 8000]--help: Show this message and exit.
harnesslab suite¶
Inspect and sanity-check task suites.
Usage:
Options:
--help: Show this message and exit.
Commands:
list: List suites, their tasks and variants.check: Sanity-check verifiers: reference...mine: Mine verifier-backed tasks from a...
harnesslab suite list¶
List suites, their tasks and variants.
Usage:
Arguments:
target: A suite.yaml, a directory, or a bundled suite name (default: bundled suites and ./suites).
Options:
--help: Show this message and exit.
harnesslab suite check¶
Sanity-check verifiers: reference solutions must pass, the untouched repo must fail.
Usage:
Arguments:
target: suite.yaml to validate, or a bundled suite name. [required]
Options:
--help: Show this message and exit.
harnesslab suite mine¶
Mine verifier-backed tasks from a repository's git history.
Every commit that changes source code and its tests becomes a candidate task: start at the parent, the commit's tests are the hidden verifier, its source change is the reference solution, its message is the prompt. Candidates are kept when the tests fail at the parent and pass at the commit.
Usage:
Arguments:
repo: A local git repository to mine (only read). [required]
Options:
-o, --out <path>: Directory to write the suite to. [required]--rev <str>: Mine commits reachable from this ref. [default: HEAD]--max-commits <int range>: Newest non-merge commits to scan. [default: 200; x>=1]--max-tasks <int range>: Stop after this many kept tasks. [x>=1]--test-command <str>: Verifier command; {tests} becomes the commit's test files (shell-quoted). [default: python -m pytest -q {tests}]--test-glob <str>: Glob marking test files (repeatable; replaces defaults).--ignore-glob <str>: Glob for files that are neither tests nor source, e.g. docs (repeatable; replaces defaults).--max-files <int range>: Skip commits changing more source files. [default: 6; x>=1]--max-lines <int range>: Skip commits changing more source lines. [default: 400; x>=1]--setup <str>: Setup command run before the tests (repeatable).--prompt-template <path>: Text file with a {message} placeholder.--validate / --no-validate: Keep only commits whose tests fail at the parent and pass at the commit. [default: validate]--timeout <int range>: Seconds per setup or test command. [default: 300; x>=1]-p, --parallelism <int range>: [default: 4; x>=1]--name <str>: Suite name.--force: Replace a previous suite in --out.--help: Show this message and exit.
harnesslab experiment¶
Inspect and export past experiments.
Usage:
Options:
--help: Show this message and exit.
Commands:
list: List past experiments.show: Print a terminal summary of an experiment.compare: Paired, task-level comparison of two...export: Export an experiment (spec, runs, traces,...
harnesslab experiment list¶
List past experiments.
Usage:
Options:
--help: Show this message and exit.
harnesslab experiment show¶
Print a terminal summary of an experiment.
Usage:
Arguments:
experiment_id: [required]
Options:
--help: Show this message and exit.
harnesslab experiment compare¶
Paired, task-level comparison of two variants: bootstrap intervals and a sign test.
Usage:
Arguments:
experiment_id: [required]a: Baseline variant id (A). [required]b: Variant compared against A (B). [required]
Options:
--resamples <int range>: [default: 2000; x>=100]--seed <int>: [default: 0]--min-tasks <int range>: [default: 5; x>=1]--help: Show this message and exit.
harnesslab experiment export¶
Export an experiment (spec, runs, traces, verdicts, aggregates) as JSON.
Usage:
Arguments:
experiment_id: [required]
Options:
--events / --no-events: Include normalized events. [default: events]--artifacts / --no-artifacts: Inline text artifacts (diffs, verifier output). [default: artifacts]-o, --output <path>: Write to a file instead of stdout.--help: Show this message and exit.
harnesslab sweep¶
Configuration sweeps: search model x effort x toolset x compaction x action policy for the cheapest verified configuration.
Usage:
Options:
--help: Show this message and exit.
Commands:
list: List bundled sweep templates and...run: Run every configuration of a sweep and...report: Recompute the recommendation of a past...
harnesslab sweep list¶
List bundled sweep templates and ./sweeps/*.yaml.
Usage:
Options:
--help: Show this message and exit.
harnesslab sweep run¶
Run every configuration of a sweep and report the cheapest verified one per workload.
Usage:
Arguments:
target: A sweep.yaml or a bundled sweep name (e.g. demo-fake). [required]
Options:
-r, --repetitions <int range>: [x>=1]-p, --parallelism <int range>: [x>=1]-n, --name <str>: Experiment name.--keep-worktrees--pricing <path>: pricing.yaml for cost estimates.--plugin <str>: Python module registering custom runners.--dry-run: Print the expanded configurations and exit.--help: Show this message and exit.
harnesslab sweep report¶
Recompute the recommendation of a past sweep from the database.
Usage:
Arguments:
experiment_id: [required]
Options:
--help: Show this message and exit.
harnesslab grow¶
Growing Harness: grow a harness bundle from failures with a held-out gate.
Usage:
Options:
--help: Show this message and exit.
Commands:
list: List grow sessions and bundled grow...run: Grow a harness: failures -> optimizer ->...resume: Continue an interrupted or failed grow...show: Print the version lineage and gate results...export: Copy the session's current accepted bundle...
harnesslab grow list¶
List grow sessions and bundled grow templates.
Usage:
Options:
--help: Show this message and exit.
harnesslab grow run¶
Grow a harness: failures -> optimizer -> window check -> gate check -> accept or roll back.
Usage:
Arguments:
target: A grow.yaml or a bundled grow name (e.g. demo-fake). [required]
Options:
--max-iterations <int range>: [x>=1]-n, --name <str>: Session name.--keep-worktrees--pricing <path>: pricing.yaml for cost estimates.--plugin <str>: Python module registering custom runners or optimizers.--dry-run: Print the split, window and first optimizer view; run nothing.--help: Show this message and exit.
harnesslab grow resume¶
Continue an interrupted or failed grow session from its saved state.
Usage:
Arguments:
session_id: [required]
Options:
--help: Show this message and exit.
harnesslab grow show¶
Print the version lineage and gate results of a grow session.
Usage:
Arguments:
session_id: [required]
Options:
--help: Show this message and exit.
harnesslab grow export¶
Copy the session's current accepted bundle to a directory and write lineage.json.
Usage:
Arguments:
session_id: [required]directory: Directory to write the current bundle into. [required]
Options:
--help: Show this message and exit.
harnesslab harness¶
Inspect and validate harness bundles.
Usage:
Options:
--help: Show this message and exit.
Commands:
check: Validate a harness bundle and, with...
harnesslab harness check¶
Validate a harness bundle and, with --suite, lint it for hidden-test leaks.
Usage:
Arguments:
bundle_dir: Harness bundle directory. [required]
Options:
--suite <str>: Suite (path or bundled name) to run the leak lint against.--help: Show this message and exit.
harnesslab ablate¶
Component ablation: test every part of a harness bundle against its own absence.
Usage:
Options:
--help: Show this message and exit.
Commands:
run: Run the full bundle, an empty (minimal)...report: Recompute the component verdicts of a past...
harnesslab ablate run¶
Run the full bundle, an empty (minimal) bundle and one leave-one-out bundle per component.
Usage:
Arguments:
bundle_dir: Harness bundle directory to ablate. [required]
Options:
--suite <str>: Suite path or bundled suite name. [required]--variant <str>: Base variant (runner, model, options) the bundle is applied to. [default: claude-default]-t, --tasks <str>: Comma-separated task ids (default: all).-r, --repetitions <int range>: [default: 2; x>=1]-p, --parallelism <int range>: [default: 2; x>=1]-n, --name <str>: Experiment name.--min-tasks <int range>: [default: 5; x>=1]--resamples <int range>: [default: 2000; x>=100]--seed <int>: [default: 0]--keep-worktrees--pricing <path>: pricing.yaml for cost estimates.--plugin <str>: Python module registering custom runners.--dry-run: Print the components and variants, run nothing.--help: Show this message and exit.
harnesslab ablate report¶
Recompute the component verdicts of a past ablation from the database.
Usage:
Arguments:
experiment_id: [required]
Options:
--help: Show this message and exit.