State: GitHub issues
A study or plan parent with code and run sub-issues. The repo holds no plan, progress or notes files. See State.
Overview
A state machine protocol for running controlled ML experiments with coding agents. You write the study spec: which variables differ between arms, which are held fixed, and which may adapt. Agents implement and run the study through GitHub issues and pull requests, one state at a time. Every run records its resolved config, and a script checks it against the spec and reports any difference the spec does not allow. You decide at four points: publish, merge, budget and conclude.
The problem: agents change fixed settings
Section titled “The problem: agents change fixed settings”Coding agents can now write the code, launch the runs and post the results of a study faster than you can read them. These changes happen below the plan, in code, config and launches. A learning rate gets changed to fix a crash. An effective batch gets halved to fit memory. Each change is reasonable on its own, and together they turn a comparison into a guess. Usually all you see is the agent’s summary.
What CVSM does about it
Section titled “What CVSM does about it”CVSM asks you for four decisions and makes the rest checkable.
- You approve a study spec: the arms, the settings held fixed (each with a reason), and the settings that may adapt. See The study spec.
- Agents do everything below it through GitHub issues and pull requests. They stop only at four gates, or at a blocker they cannot resolve alone.
- Every run records its resolved config, and the results command checks those records against the spec. Deviations reach you as facts taken from the records. See Run records.
What you copy in
Section titled “What you copy in”CVSM is a set of files and conventions around git and GitHub. Nothing is installed, and nothing runs outside GitHub, git and your own machines.
Rules: AGENTS.md
How to read state, when to stop, how to ask, how to use git and files. At most 80 lines, capped by a test. See AGENTS.md.
Procedures: four skills
plan, implement, review and run, each at most 120 lines. Claude Code and Codex read the same files. See Skills.
Checks: guard, test, merge.py
guard checks each PR’s format, file locations and scope; test runs your tests; merge.py is the only way an agent merges. See Merging.
Runs: run.py and results.py
A reference Run command that records each run as an MLflow run, and a results command that builds the table and the spec check. See run.py.
Review: a fresh session
Every PR is reviewed by a session that starts from the PR number alone. See Review.
CVSM in numbers
Section titled “CVSM in numbers”| Services to install | 0 |
| Lines of rules (AGENTS.md) | at most 80, checked by a test |
| Skills | 4, at most 120 lines each |
| Human gates | 4, the only planned stops |
| Merge paths for agents | 1, merge.py |
| Agent CLIs | 2, Claude Code and Codex, with one set of skills |
What CVSM leaves to you
Section titled “What CVSM leaves to you”CVSM records facts and shows differences. It does not choose seeds, design experiments, decide when to rerun, or judge whether runs are comparable; you set those in the spec and in your instructions. Analysis, visualization, and how an agent works inside a task are also yours. See Principles.
Status
Section titled “Status”CVSM v0.1.0 is a pre-release. It runs its own development, and its interfaces may still change before 1.0. See Status.