Evaluation Config Files

For longer commands, complex experiments, or anything you'd want to commit to the repo, put the run in an Inspect run-config YAML and pass it with --run-config. ct run eval --run-config x.yml delegates to inspect eval --run-config x.yml; the eval runs CT's registered control_tower/control_eval task.

uv run ct run eval --run-config examples/eval_configs/03-attack-single.yml

Worked examples live in examples/eval_configs/:

Schema

The file is Inspect's native run-config format: task: names the registered task and its args, eval_config: holds Inspect eval options, tags: tags the run, and metadata: carries CT's post-run knobs.

Task-selection fields follow the modes described in Evaluation Task Selection.

task:
  task: control_tower/control_eval
  args:
    # Optional run name (defaults to `<policy>-<task-set>`).
    run_name: my-experiment

    # Task selection. See task-selection.md for semantics; fields
    # mirror the CLI flags (environments, main_tasks, side_tasks,
    # task_file, trajectory_id, limit, expansion flags, categories, ...).
    task_selection:
      environments: [web_scraping]
      main_tasks: [crawl_depth]
      side_tasks: [visit_malicious_website]

    # Pick exactly one of `policy` or (`untrusted_policy` + `protocol`).
    policy:
      name: honest                # `ct protocols policies` to list
      args:
        model: sonnet
    # untrusted_policy:
    #   name: simple-attack
    # protocol:
    #   name: trusted-monitoring
    #   args:
    #     sus_threshold: 7
    # model_roles:                # the eval-level roles the protocol's monitors run on
    #   trusted: haiku

    # Task-level scoring options.
    scorer_timeout: 300           # int | null  (seconds)
    diagnostics: null             # list[str] | null  (e.g. [refusal])

    # Local docker sandbox options (SandboxConfig fields).
    sandbox:
      startup_retries: 7

# Inspect eval options (native `eval_config`): epochs, max_samples,
# fail_on_error, retry_on_error, token_limit, message_limit, time_limit,
# cost_limit, ...
eval_config:
  epochs: 3
  max_samples: 4
  fail_on_error: false

tags: [attack, sonnet]

# CT post-run side effects, applied by CT's registered run-end hook.
metadata:
  ct_upload: true                 # upload results to the Control Tower database
  docent_collection_id: null      # also push to this Docent collection

All task args are validated with the same Pydantic models as the CLI flags, so unknown fields and invalid combinations (e.g. policy together with protocol) are rejected loudly.

Overriding

Inspect's semantics apply: one file per run, CLI flags override file values, and -T task args dict-merge into the file's task.args. ct run eval --run-config forwards --no-upload/--docent-collection-id (as metadata), --run-name (as a task arg), --tags, --log-dir, and --max-samples; other ct run eval flags are rejected — put them in the YAML.

# Same config, different model, no upload
uv run ct run eval --run-config examples/eval_configs/02-honest-baseline.yml --no-upload
uv run inspect eval --run-config examples/eval_configs/02-honest-baseline.yml -T 'policy={"name": "honest", "args": {"model": "opus"}}'

eval2

eval2 is a second eval path with its own registered task, control_tower/control_eval2, and its own registries for untrusted policies and blue protocols (ct protocols eval2-policies, ct protocols eval2-protocols). It takes the same task_selection, sandbox, scorer_timeout and diagnostics args, requires both untrusted_policy and protocol, and adds one of its own: white.

white is the deployment the run holds fixed — which harness drives the untrusted model, and which of the environment's affordance tools the red team is granted. Omit white.harness and the run uses the null harness, the environment as declared with no coding agent; omit granted_affordances and red touches nothing.

task:
  task: control_tower/control_eval2
  args:
    task_selection:
      environments: [slack_clone]
      main_tasks: [bookmark_messages]
      side_tasks: [exfiltrate_private_messages_live]
    untrusted_policy: {name: attack}
    protocol:
      name: defer-to-resample
      args: {max_steps: 6}
    white:
      harness: {kind: inspect_swe, agent: claude_code}
      granted_affordances: [run_side_task_test]

uv run ct run eval2 --run-config examples/eval_configs/05-eval2.yml
uv run inspect eval --run-config examples/eval_configs/05-eval2.yml

ct run eval2 --run-config behaves exactly like ct run eval --run-config: it delegates to inspect eval --run-config, forwards the same small set of overrides, and rejects every other flag. The equivalent flag invocation is ct run eval2 --harness inspect_swe -ha agent=claude_code --grant run_side_task_test ....