Run experiments

After you publish a study, agents implement its code tasks, launch its runs within budget, and post results comments generated from the run records. You step in at the merge, budget and conclude gates, and at blockers.

1. Get the code merged

Say “do #12” for a code task, or “take #10 to analysis” to have the agent work through the whole study. Under a study, the spec binds the code: every key it names is set through config, the resolved config is written to $RUN_DIR/config.json from one process, and nothing changes a Controlled setting or makes the arms differ beyond Arms. The fresh-context reviewer checks the same, and a violation blocks the PR.

2. Launch the runs

A run issue becomes ready once the code issues it is blocked by are closed. The agent claims it, makes a clean worktree from main and launches every Runs line from there, so each record names merged code. A run issue has no PR.

Before launching, it checks that each Runs line runs exactly one arm and that each arm the issue runs gets every value of each Controlled in line. Each line goes through the Run command on the Run line of AGENTS.md:

<run> --issue N -- <cmd>
<run> --issue N --gpus 4 -- <cmd>   # for the Runs line `--gpus 4 -- <cmd>`

The reference Run command, scripts/run.py, records each run as an MLflow run and waits for free GPUs, so several lines can be launched at once. Runs already recorded for the issue count, so a resumed session launches only the lines with no record yet.

3. Check right after launch

As soon as the runs have logged their configs as cfg.* params, the agent runs the results command with the study’s spec:

gh issue view P --json body -q .body | <results> --issue N --spec -

An Adaptable setting may change within its rule, for example a smaller per-device batch after an out-of-memory error. The agent notes each change, its run and the reason for the results comment.

4. Budget and failures

GPU hours used is the sum, over the issue’s runs (failed ones included), of wall time times the number of GPUs. The agent estimates the rest from the first finished run.

A failed run stays recorded and is counted (失败数 in the table).
A line is rerun only after an infra failure: a lost node or GPU, preemption, a full disk, a lost connection to the tracking backend. Never because a result looks wrong.
A failure caused by the code is a result, not infra. The agent keeps it, opens a code issue for the bug and goes on with the other lines. Whether to run again after the fix is your call.
The same failure 3 times in a row is a blocker.

5. Results comments

When every line has a finished or failed record, the agent posts one comment on the run issue: a short note (what ran, failures and reruns with their causes, every Adaptable change, and whether the budget was exceeded), followed by the output of the results command with --metric and --spec, pasted unedited. No run is dropped, filtered or reordered. Then it closes the run issue.

When all run issues of the study are closed, it posts the same table and spec check over all of them on the study issue: the study table. The study now awaits analysis.