CVSMPre-release

01 — Overview

CVSM

Controlled-VariableState Machine Protocol

A state machine protocol for running controlled ML experiments with coding agents. You write the study spec: which variables differ between arms, which are held fixed, and which may adapt. Agents implement and run the study through GitHub issues and pull requests, one state at a time. Every run records its resolved config, and a script checks it against the spec and reports any difference the spec does not allow. You decide at four points: publish, merge, budget and conclude.

Read the docsQuick startGitHub
Human gateAgent workChecked against the specdoem97/cvsm

02 — Problem

The spec and the layers below it

Agents change settingsthat a comparison needs fixed

Coding agents now write the code, launch the runs and post the results of an ML study faster than you can read them.

A learning rate gets changed to fix a crash. An effective batch gets halved to fit memory. Each change is reasonable on its own; together they turn a comparison into a guess. Usually all you see is the agent’s summary.

The study spec is the one layer you can read in full, so CVSM checks every run against it, from the run’s own record.

03 — State machine

Four gates · every step on GitHub

A study is astate machine on GitHub

A study moves through plan, implement, review, merge, run and analyze, and each step is recorded on GitHub. Each instruction names a target state. The agent keeps going until it reaches that state, a gate or a blocker, and it stops at the four gates even when everything works.

Read more
Humandecides in a session
Agentsfour skills; Claude Code or Codex
RecordGitHub and MLflow, written as it happens

01 Plan

Human
Agents
plandiscuss, draft
Record

Issues

  • study #10
  • code #11
  • run #12
  • Decision:

02 Implement

Human
Agents
implementcode to a PR
Record

Pull request

  • Closes #11
  • guard · test
  • Decision:

03 Review

Human
Agents
reviewfresh session
Record

Review

  • Verdict: pass
  • @ 3f9c2a1

04 Merge

Human
Agents
merge.pythe only merge
Record

main

  • squash merge
  • of that head
  • Decision:

05 Run

Human
Agents
runlaunch, check
Record

MLflow runs

  • cvsm.* cfg.*
  • artifacts
  • Decision:

06 Analyze

Human
Agents
reportthree lines
Record

Results

  • per-issue table
  • spec check
  • study table
  • Decision:

01Publish

When
New or changed approved work: publishing a plan; changing a task’s Goal, Scope, Acceptance or Merge, or a study’s Arms, Controlled or Adaptable; changing a protected path.
You say
“publish”, or approve the change
Recorded
The parent and its sub-issues, created wave by wave, and a Decision: comment on the parent.

Blocker

When
The same failure 3 times in a row, 3 changes verdicts in a row, an ambiguity that would change the deliverable, or a Controlled setting that can’t be met without changing the comparison.
You say
one of at most 3 options, e.g. “option 2”
Recorded
needs-decision and a four-line comment (Blocked:, Evidence:, Options:, Recommend:) when you’re away; then your Decision: comment.

02Merge

When
A PR is merge-ready: guard and test green and Verdict: pass on the current head. Under Merge: agent the agent merges it itself with merge.py, which refuses protected paths and unreviewed heads.
You say
“merge”, or “merge #15”
Recorded
merge.py --human squash-merges that head; the task closes, with a Decision: comment.

03Budget

When
The next step would exceed the budget in tokens, time or GPU hours.
You say
“continue”, with a new budget if you like
Recorded
A Decision: comment with the budget you set.

04Conclude

When
All of a study’s sub-issues are closed and it awaits analysis.
You say
your conclusion, e.g. “conclude #10: …”
Recorded
The study closes with your conclusion, recorded as a Decision: comment.

Hover or focus a gate

04 — Spec check

Study spec and run records

Every run is checkedagainst the study spec

You approve a study spec at gate 01: the arms, the settings held fixed (each with a reason), and what may adapt. Every run records its resolved config, and the results command checks those records against the spec, right after launch and in every results comment.

Read more
Study #10 · specReadable by you · gate 01
  1. ### Armsthe only differences
  2. full: model.adapter=none
  3. lora: model.adapter=lora model.lora.rank=8
  4. ### Controlledheld fixed, each with a reason
  5. optim.lr=3e-4# paper X, Table 2
  6. train.effective_batch=256# paper X, Table 2
  7. seed in 1,2,3# three seeds, as in paper X
  8. ### Adaptablemay differ, within its rule
  9. train.per_device_batch# fit to GPU memory; effective batch stays 256
  10. train.accum# follows per_device_batch
spec checkfrom the records · 6 runs

规格核对：

  • ≠train.effective_batch=256 不成立：1 个 run，0330bc8a…（128）The effective-batch line fails for one run, which took 128.
  • ≠未声明的差异 train.precision：bf16（5 个 run）、fp16（1 个 run）train.precision differs between runs, although no section declares it.
  • ·Adaptable train.per_device_batch：full 取 64（3 个 run）；lora 取 64（2 个 run）、32（1 个 run）The values each Adaptable key took, per arm.
  • ·Adaptable train.accum：full 取 4（3 个 run）；lora 取 4（3 个 run）

Output of scripts/results.py

It lists, as facts

  1. problems of the spec itself: unreadable lines, Controlled lines without a reason, missing sections, keys in several sections
  2. runs where a Controlled line does not hold, or whose config never recorded its key
  3. runs that match no arm, or several
  4. arms missing a value of an in line
  5. keys in no section whose values differ: undeclared differences
  6. arms with no run
  7. the values each Adaptable key took per arm

CVSM reports what the records show. Whether the runs are still comparable is your call.

05 — Decisions

Claude Code or Codex

You decide in a sessionat four points

You make decisions only in an agent session. Agents create issues, open PRs, review, merge and close on your instruction, and record each of your decisions on the issue.

Read more
  1. You

    let’s plan this study: does LoRA match full fine-tuning?

  2. Agent

    Discusses the spec with you, then drafts study #10 with code #11 and run #12.

  3. You

    publish

    Gate 01
  4. Agent

    Done:published #10 with #11 and #12; Decision on #10

    Waiting for you:nothing

    Next:#11 is ready

  5. You

    do #11

  6. Agent

    Done:PR #13 is merge-ready, Verdict: pass @ 3f9c2a1

    Waiting for you:merge PR #13

    Next:run #12

  7. You

    merge

    Gate 02
  8. Agent

    Done:merged PR #13 with merge.py --human; #11 closed; Decision on #11

    Waiting for you:nothing

    Next:run #12

  9. You

    take #10 to analysis

  10. Agent

    Done:#12 closed with its results and spec check; study table on #10

    Waiting for you:conclude #10 (gate 04)

    Next:nothing until you decide

Every stop ends with three linesDone:Waiting for you:Next:

06 — Contents

Six numbers

What the protocolconsists of

  • 0services to install
  • 80lines of rules, capped by a test
  • 4skills of at most 120 lines each
  • 4human gates, the only planned stops
  • 1merge path for agents: merge.py
  • 2agent CLIs, Claude Code and Codex, with one set of skills

07 — Adopt

Pre-release

Copy the protocolinto your repo

Copy only tracked files, then fill in the Project section of AGENTS.md. An agent in your session can do every step; you decide, and lend an admin account where one is needed.

git -C <cvsm clone> archive HEAD AGENTS.md .agents .claude .codex .github scripts tests ':!.github/workflows/pages.yml' | tar -x -C <your repo>
Adopt CVSM in five stepsQuick start