From 30ac7a8114500a7fc24f57c4d539dfb9dbbf48e4 Mon Sep 17 00:00:00 2001 From: meelstorm Date: Fri, 17 Jul 2026 17:02:15 +0000 Subject: [PATCH] Add Plan 2: Python PPO training pipeline plan --- .../plans/2026-07-17-02-training-pipeline.md | 789 ++++++++++++++++++ 1 file changed, 789 insertions(+) create mode 100644 docs/superpowers/plans/2026-07-17-02-training-pipeline.md diff --git a/docs/superpowers/plans/2026-07-17-02-training-pipeline.md b/docs/superpowers/plans/2026-07-17-02-training-pipeline.md new file mode 100644 index 0000000..601faaa --- /dev/null +++ b/docs/superpowers/plans/2026-07-17-02-training-pipeline.md @@ -0,0 +1,789 @@ +# Training Pipeline Implementation Plan + +> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking. + +**Goal:** Build the Python side of the design: a `LanderCliEnv` Gymnasium environment that wraps the `GameCli` binary as a subprocess, a `train.py` that trains a PPO policy on N vectorized copies of it, and an `export_onnx.py` that exports the trained network as `models/ppo_lander.onnx` for the ASP.NET backend to load. Reference spec at `docs/superpowers/specs/2026-07-17-cursor-following-lander-design.md`. + +**Architecture:** All Python code lives under `Training/`. `LanderCliEnv` speaks the exact protocol the `GameCli` documents on stdio, so training uses literally the same physics as inference. Training runs a `SubprocVecEnv` of N envs → PPO batches transitions → checkpoints to `checkpoints/` → export → verify → `models/ppo_lander.onnx`. + +**Tech Stack:** Python 3.11 or 3.12, `gymnasium`, `stable-baselines3`, `torch`, `onnx`, `onnxruntime`, `pytest`. + +--- + +## Prerequisites + +- **Plan 1 complete.** The `GameCli` binary must be publishable at `publish/GameCli/GameCli` (run `dotnet publish GameCli/GameCli.csproj -c Release -o publish/GameCli` if it's not there yet). All Training/ code assumes that path. +- Python 3.11 or 3.12 available as `python3`. + +--- + +## File structure + +``` +Experiment_ReinforcementLearning/ +├── Training/ +│ ├── requirements.txt +│ ├── pytest.ini +│ ├── lander_cli_env.py # LanderCliEnv(gym.Env) +│ ├── train.py # PPO training with SubprocVecEnv +│ ├── export_onnx.py # SB3 -> ONNX +│ └── tests/ +│ ├── __init__.py +│ ├── test_lander_cli_env.py +│ └── test_export_onnx.py +├── models/ # gitignored; export_onnx.py writes ppo_lander.onnx here +├── checkpoints/ # gitignored; SB3 checkpoints +└── tensorboard/ # gitignored; TB logs +``` + +`Training/lander_cli_env.py` is the single file that touches the subprocess. `train.py` and `export_onnx.py` just consume Gymnasium/SB3 APIs. + +--- + +## Task 1: Scaffold Python project + +**Files:** +- Create: `Training/requirements.txt` +- Create: `Training/pytest.ini` +- Create: `Training/tests/__init__.py` + +- [ ] **Step 1: Check Python is available** + +Run: `python3 --version` +Expected: `Python 3.11.x` or `3.12.x`. If older or missing, STOP and report BLOCKED. + +- [ ] **Step 2: Create the virtualenv at repo root** + +```bash +python3 -m venv .venv +source .venv/bin/activate +pip install --upgrade pip +``` + +- [ ] **Step 3: Write `Training/requirements.txt`** + +``` +gymnasium>=0.29,<2 +stable-baselines3>=2.3,<3 +torch>=2.2,<3 +onnx>=1.16,<2 +onnxruntime>=1.18,<2 +numpy>=1.24,<3 +tensorboard>=2.15 +pytest>=8 +``` + +- [ ] **Step 4: Install dependencies** + +Run: `pip install -r Training/requirements.txt` +Expected: successful install. This may take a few minutes (torch is large). + +- [ ] **Step 5: Write `Training/pytest.ini`** + +```ini +[pytest] +testpaths = tests +python_files = test_*.py +addopts = -v +``` + +- [ ] **Step 6: Write `Training/tests/__init__.py`** (empty file) + +```bash +touch Training/tests/__init__.py +``` + +- [ ] **Step 7: Verify pytest can discover an empty test suite** + +From `Training/` directory: + +```bash +cd Training && pytest && cd .. +``` + +Expected: `no tests ran` (0 tests, exit code 0 or 5 — both mean "no tests but no failures"). Exit code 5 is fine. + +- [ ] **Step 8: Commit** + +```bash +git add Training/ +git -c user.email=meelstorm@gmail.com -c user.name=meelstorm commit -m "feat(training): scaffold Python training project" +``` + +--- + +## Task 2: `LanderCliEnv` — Gymnasium environment wrapping the CLI + +**Files:** +- Create: `Training/lander_cli_env.py` +- Create: `Training/tests/test_lander_cli_env.py` + +### Protocol quick reference (from Plan 1) + +- CLI startup emits: `{"init":{"world":[1,1],"dt":0.02,"obs_dim":7,"n_actions":4}}` +- Step input: `{"action":,"target":[,]}` +- Reset input: `{"cmd":"reset","seed":}` (seed optional) +- Output line: `{"obs":[...7 floats...],"state":{...},"reward":,"done":,"step":}` +- CLI binary path (relative to repo root): `publish/GameCli/GameCli` + +- [ ] **Step 1: Write the failing tests** + +Create `Training/tests/test_lander_cli_env.py`: + +```python +"""Unit tests for LanderCliEnv.""" +from pathlib import Path + +import numpy as np +import pytest +from gymnasium.utils.env_checker import check_env + +# Add parent dir to sys.path so we can import lander_cli_env +import sys +sys.path.insert(0, str(Path(__file__).resolve().parent.parent)) + +from lander_cli_env import LanderCliEnv + + +REPO_ROOT = Path(__file__).resolve().parents[2] +CLI_BINARY = REPO_ROOT / "publish" / "GameCli" / "GameCli" + + +@pytest.fixture(scope="module") +def cli_binary_exists(): + if not CLI_BINARY.exists(): + pytest.skip(f"CLI binary not found at {CLI_BINARY} — run `dotnet publish GameCli -c Release -o publish/GameCli`") + return CLI_BINARY + + +def test_env_opens_and_closes(cli_binary_exists): + env = LanderCliEnv(cli_binary=str(cli_binary_exists)) + env.close() # should not raise + + +def test_observation_space_is_seven_dim(cli_binary_exists): + env = LanderCliEnv(cli_binary=str(cli_binary_exists)) + try: + assert env.observation_space.shape == (7,) + finally: + env.close() + + +def test_action_space_is_discrete_four(cli_binary_exists): + env = LanderCliEnv(cli_binary=str(cli_binary_exists)) + try: + assert env.action_space.n == 4 + finally: + env.close() + + +def test_reset_returns_seven_dim_observation(cli_binary_exists): + env = LanderCliEnv(cli_binary=str(cli_binary_exists)) + try: + obs, info = env.reset(seed=42) + assert isinstance(obs, np.ndarray) + assert obs.shape == (7,) + assert obs.dtype == np.float32 + finally: + env.close() + + +def test_step_advances_and_returns_five_tuple(cli_binary_exists): + env = LanderCliEnv(cli_binary=str(cli_binary_exists)) + try: + env.reset(seed=0) + obs, reward, terminated, truncated, info = env.step(0) # noop + assert obs.shape == (7,) + assert isinstance(reward, float) + assert terminated is False # never terminates + assert isinstance(truncated, bool) + finally: + env.close() + + +def test_reset_randomizes_target_across_episodes(cli_binary_exists): + env = LanderCliEnv(cli_binary=str(cli_binary_exists)) + try: + # Two resets with different seeds should generally give different targets. + env.reset(seed=1) + target1 = env._target + env.reset(seed=2) + target2 = env._target + assert target1 != target2 + finally: + env.close() + + +def test_truncation_at_max_steps(cli_binary_exists): + env = LanderCliEnv(cli_binary=str(cli_binary_exists), max_episode_steps=5) + try: + env.reset(seed=0) + truncated = False + for _ in range(5): + _, _, _, truncated, _ = env.step(0) + assert truncated is True + finally: + env.close() + + +def test_check_env_passes(cli_binary_exists): + """Gymnasium's own env sanity checker — spaces, dtypes, reset/step contract.""" + env = LanderCliEnv(cli_binary=str(cli_binary_exists), max_episode_steps=50) + try: + # skip_render_check because we don't implement rendering + check_env(env.unwrapped, skip_render_check=True) + finally: + env.close() +``` + +- [ ] **Step 2: Run tests to verify they fail** + +```bash +cd Training && pytest && cd .. +``` + +Expected: import error — `lander_cli_env` module does not exist. + +- [ ] **Step 3: Write `Training/lander_cli_env.py`** + +```python +"""Gymnasium environment that wraps the GameCli C# binary as a subprocess. + +Speaks the line-delimited JSON protocol defined in: + docs/superpowers/specs/2026-07-17-cursor-following-lander-design.md + +Each episode picks a random target in [0,1]^2 at reset time and holds it fixed. +Never terminates — episodes end via truncation at max_episode_steps. +""" +from __future__ import annotations + +import json +import subprocess +from pathlib import Path +from typing import Any, Optional + +import gymnasium as gym +import numpy as np +from gymnasium import spaces + + +class LanderCliEnv(gym.Env): + """One CLI subprocess per env instance. Not thread-safe; safe under SubprocVecEnv.""" + + metadata = {"render_modes": []} + + def __init__( + self, + cli_binary: str, + max_episode_steps: int = 1000, + ) -> None: + super().__init__() + self._cli_binary = cli_binary + self._max_episode_steps = max_episode_steps + + self._proc: Optional[subprocess.Popen] = None + self._target: tuple[float, float] = (0.5, 0.5) + self._step_count: int = 0 + + self._start_process() + self._read_init() + + self.observation_space = spaces.Box( + low=-np.inf, high=np.inf, shape=(7,), dtype=np.float32 + ) + self.action_space = spaces.Discrete(4) + + # -------- subprocess lifecycle -------- + + def _start_process(self) -> None: + self._proc = subprocess.Popen( + [self._cli_binary], + stdin=subprocess.PIPE, + stdout=subprocess.PIPE, + stderr=subprocess.PIPE, + text=True, + bufsize=1, # line-buffered + ) + + def _read_init(self) -> None: + assert self._proc is not None and self._proc.stdout is not None + line = self._proc.stdout.readline() + if not line: + raise RuntimeError("CLI died before emitting init handshake") + msg = json.loads(line) + if "init" not in msg: + raise RuntimeError(f"expected init handshake, got: {line!r}") + + def _send(self, obj: dict[str, Any]) -> None: + assert self._proc is not None and self._proc.stdin is not None + self._proc.stdin.write(json.dumps(obj) + "\n") + self._proc.stdin.flush() + + def _recv(self) -> dict[str, Any]: + assert self._proc is not None and self._proc.stdout is not None + line = self._proc.stdout.readline() + if not line: + raise RuntimeError("CLI closed stdout unexpectedly") + return json.loads(line) + + # -------- gym.Env API -------- + + def reset( + self, *, seed: Optional[int] = None, options: Optional[dict] = None + ) -> tuple[np.ndarray, dict]: + super().reset(seed=seed) + # Sample a fresh target each episode; the Gymnasium-provided + # np_random is deterministic given `seed`. + tx = float(self.np_random.uniform(0.05, 0.95)) + ty = float(self.np_random.uniform(0.05, 0.95)) + self._target = (tx, ty) + self._step_count = 0 + + cmd: dict[str, Any] = {"cmd": "reset"} + if seed is not None: + cmd["seed"] = int(seed) + self._send(cmd) + msg = self._recv() + obs = np.asarray(msg["obs"], dtype=np.float32) + # Overwrite the CLI's dummy (0.5, 0.5) target displacement with the real one. + # dx = target.X - ship.X, dy = target.Y - ship.Y. + obs[0] = tx - msg["state"]["x"] + obs[1] = ty - msg["state"]["y"] + return obs, {} + + def step(self, action: int) -> tuple[np.ndarray, float, bool, bool, dict]: + self._send({"action": int(action), "target": list(self._target)}) + msg = self._recv() + obs = np.asarray(msg["obs"], dtype=np.float32) + reward = float(msg["reward"]) + self._step_count += 1 + truncated = self._step_count >= self._max_episode_steps + terminated = False # continuous hover task + return obs, reward, terminated, truncated, {} + + def close(self) -> None: + if self._proc is None: + return + try: + if self._proc.stdin is not None: + self._proc.stdin.close() + except (BrokenPipeError, OSError): + pass + try: + self._proc.wait(timeout=3) + except subprocess.TimeoutExpired: + self._proc.kill() + self._proc.wait(timeout=1) + self._proc = None + + def __del__(self) -> None: + try: + self.close() + except Exception: + pass +``` + +- [ ] **Step 4: Publish the CLI binary (prerequisite for tests)** + +If `publish/GameCli/GameCli` doesn't exist yet, run: + +```bash +dotnet publish GameCli/GameCli.csproj -c Release -o publish/GameCli +``` + +- [ ] **Step 5: Run tests to verify they pass** + +```bash +cd Training && pytest tests/test_lander_cli_env.py && cd .. +``` + +Expected: 8 passed, 0 failed. + +- [ ] **Step 6: Commit** + +```bash +git add Training/lander_cli_env.py Training/tests/test_lander_cli_env.py +git -c user.email=meelstorm@gmail.com -c user.name=meelstorm commit -m "feat(training): implement LanderCliEnv wrapping the GameCli subprocess" +``` + +--- + +## Task 3: `train.py` — PPO training with vectorized envs + +**Files:** +- Create: `Training/train.py` + +- [ ] **Step 1: Write `Training/train.py`** + +```python +"""Train a PPO policy on LanderCliEnv. + +Usage: + python train.py --steps 100000 # short smoke run + python train.py --steps 2000000 --n-envs 8 # full training +""" +from __future__ import annotations + +import argparse +from pathlib import Path + +from stable_baselines3 import PPO +from stable_baselines3.common.callbacks import CheckpointCallback +from stable_baselines3.common.vec_env import SubprocVecEnv, DummyVecEnv + +# sys.path shim so this script can be run from any CWD. +import sys +sys.path.insert(0, str(Path(__file__).resolve().parent)) + +from lander_cli_env import LanderCliEnv + + +REPO_ROOT = Path(__file__).resolve().parents[1] +CLI_BINARY = REPO_ROOT / "publish" / "GameCli" / "GameCli" + + +def make_env(seed: int, max_episode_steps: int): + """Return a thunk that SubprocVecEnv can call to construct one env.""" + def _fn(): + env = LanderCliEnv( + cli_binary=str(CLI_BINARY), + max_episode_steps=max_episode_steps, + ) + env.reset(seed=seed) + return env + return _fn + + +def main() -> None: + parser = argparse.ArgumentParser() + parser.add_argument("--steps", type=int, default=2_000_000, + help="total env steps to train for") + parser.add_argument("--n-envs", type=int, default=8, + help="parallel envs; each spawns one GameCli subprocess") + parser.add_argument("--max-episode-steps", type=int, default=1000) + parser.add_argument("--seed", type=int, default=0) + parser.add_argument("--checkpoint-dir", type=Path, + default=REPO_ROOT / "checkpoints") + parser.add_argument("--tb-dir", type=Path, + default=REPO_ROOT / "tensorboard") + parser.add_argument("--use-dummy", action="store_true", + help="run envs in-process (slower, easier to debug)") + args = parser.parse_args() + + if not CLI_BINARY.exists(): + raise SystemExit( + f"CLI binary not found at {CLI_BINARY}. Run: " + f"dotnet publish GameCli/GameCli.csproj -c Release -o publish/GameCli" + ) + + args.checkpoint_dir.mkdir(parents=True, exist_ok=True) + args.tb_dir.mkdir(parents=True, exist_ok=True) + + env_fns = [ + make_env(seed=args.seed + i, max_episode_steps=args.max_episode_steps) + for i in range(args.n_envs) + ] + vec_env_cls = DummyVecEnv if args.use_dummy else SubprocVecEnv + vec_env = vec_env_cls(env_fns) + + model = PPO( + "MlpPolicy", + vec_env, + verbose=1, + seed=args.seed, + tensorboard_log=str(args.tb_dir), + policy_kwargs=dict(net_arch=[64, 64]), + ) + + checkpoint_cb = CheckpointCallback( + save_freq=max(args.steps // 10, 1) // args.n_envs, + save_path=str(args.checkpoint_dir), + name_prefix="ppo_lander", + ) + + try: + model.learn(total_timesteps=args.steps, callback=checkpoint_cb) + final_path = args.checkpoint_dir / "ppo_lander_final.zip" + model.save(str(final_path)) + print(f"saved final model to {final_path}") + finally: + vec_env.close() + + +if __name__ == "__main__": + main() +``` + +- [ ] **Step 2: Publish binary (idempotent)** + +```bash +dotnet publish GameCli/GameCli.csproj -c Release -o publish/GameCli +``` + +- [ ] **Step 3: Smoke-run training for a small step count** + +```bash +source .venv/bin/activate +cd Training && python train.py --steps 5000 --n-envs 2 --use-dummy --max-episode-steps 200 && cd .. +``` + +Expected: +- SB3 prints the standard rollout table (`| rollout/ | ...`). +- Final line: `saved final model to .../checkpoints/ppo_lander_final.zip`. +- Exit code 0. +- The file `checkpoints/ppo_lander_final.zip` exists. + +The `--use-dummy` flag runs envs in-process which is easier for debugging. Real training uses SubprocVecEnv (leave that flag off). + +- [ ] **Step 4: Commit** + +```bash +git add Training/train.py +git -c user.email=meelstorm@gmail.com -c user.name=meelstorm commit -m "feat(training): PPO training script with vectorized LanderCliEnv" +``` + +--- + +## Task 4: `export_onnx.py` — export trained policy to ONNX + +**Files:** +- Create: `Training/export_onnx.py` +- Create: `Training/tests/test_export_onnx.py` + +- [ ] **Step 1: Write the failing tests** + +Create `Training/tests/test_export_onnx.py`: + +```python +"""Verify export_onnx.py produces an ONNX file that matches the SB3 policy.""" +from pathlib import Path + +import numpy as np +import pytest + + +REPO_ROOT = Path(__file__).resolve().parents[2] +CHECKPOINT = REPO_ROOT / "checkpoints" / "ppo_lander_final.zip" +ONNX_OUT = REPO_ROOT / "models" / "ppo_lander.onnx" + + +@pytest.fixture(scope="module") +def exported_onnx(): + if not CHECKPOINT.exists(): + pytest.skip(f"no checkpoint at {CHECKPOINT} — run train.py first") + # export + import subprocess + import sys + training_dir = REPO_ROOT / "Training" + result = subprocess.run( + [sys.executable, str(training_dir / "export_onnx.py"), + "--checkpoint", str(CHECKPOINT), "--out", str(ONNX_OUT)], + capture_output=True, text=True, cwd=str(REPO_ROOT), + ) + assert result.returncode == 0, f"export failed: {result.stderr}" + assert ONNX_OUT.exists(), f"ONNX file not produced at {ONNX_OUT}" + return ONNX_OUT + + +def test_onnx_file_exists(exported_onnx): + assert exported_onnx.stat().st_size > 0 + + +def test_onnx_input_output_shapes(exported_onnx): + import onnx + model = onnx.load(str(exported_onnx)) + assert len(model.graph.input) == 1 + in_shape = [d.dim_value for d in model.graph.input[0].type.tensor_type.shape.dim] + # (batch, 7) + assert in_shape[-1] == 7 + assert len(model.graph.output) >= 1 + out_shape = [d.dim_value for d in model.graph.output[0].type.tensor_type.shape.dim] + assert out_shape[-1] == 4 # 4 discrete actions + + +def test_onnx_output_matches_sb3(exported_onnx): + """Sanity: ONNX inference for 100 random obs must argmax to the same action as SB3.""" + import onnxruntime + from stable_baselines3 import PPO + + model = PPO.load(str(CHECKPOINT)) + session = onnxruntime.InferenceSession(str(exported_onnx)) + input_name = session.get_inputs()[0].name + + rng = np.random.default_rng(1234) + obs_batch = rng.normal(size=(100, 7)).astype(np.float32) + + # ONNX: (100, 4) logits + onnx_logits = session.run(None, {input_name: obs_batch})[0] + onnx_actions = onnx_logits.argmax(axis=-1) + + # SB3 deterministic prediction + sb3_actions, _ = model.predict(obs_batch, deterministic=True) + + mismatches = int((onnx_actions != sb3_actions).sum()) + # Allow up to 2 mismatches out of 100 for float rounding on marginal states. + assert mismatches <= 2, f"{mismatches}/100 argmax mismatches between ONNX and SB3" +``` + +- [ ] **Step 2: Run tests to verify they fail** + +```bash +cd Training && pytest tests/test_export_onnx.py && cd .. +``` + +Expected: fail because `export_onnx.py` does not exist. + +- [ ] **Step 3: Write `Training/export_onnx.py`** + +```python +"""Export a trained SB3 PPO policy to ONNX for the .NET backend. + +The exported model has a single float input of shape [batch, 7] (the observation +vector) and a single float output of shape [batch, 4] (action logits). Argmax +over the last axis gives the action. +""" +from __future__ import annotations + +import argparse +from pathlib import Path + +import torch +from stable_baselines3 import PPO + + +class OnnxablePolicy(torch.nn.Module): + """Wraps the SB3 policy's actor path for ONNX export. + + Standard SB3 MlpPolicy for a discrete action space: + features = policy.extract_features(obs) + latent_pi = policy.mlp_extractor.forward_actor(features) + logits = policy.action_net(latent_pi) + """ + + def __init__(self, policy: torch.nn.Module) -> None: + super().__init__() + self.policy = policy + + def forward(self, obs: torch.Tensor) -> torch.Tensor: + features = self.policy.extract_features(obs) + latent_pi = self.policy.mlp_extractor.forward_actor(features) + return self.policy.action_net(latent_pi) + + +def main() -> None: + parser = argparse.ArgumentParser() + parser.add_argument("--checkpoint", type=Path, required=True, + help="path to SB3 .zip checkpoint") + parser.add_argument("--out", type=Path, required=True, + help="destination .onnx path") + parser.add_argument("--opset", type=int, default=17) + args = parser.parse_args() + + args.out.parent.mkdir(parents=True, exist_ok=True) + + model = PPO.load(str(args.checkpoint), device="cpu") + model.policy.eval() + + onnxable = OnnxablePolicy(model.policy) + onnxable.eval() + + dummy = torch.randn(1, 7, dtype=torch.float32) + torch.onnx.export( + onnxable, + dummy, + str(args.out), + input_names=["obs"], + output_names=["logits"], + dynamic_axes={"obs": {0: "batch"}, "logits": {0: "batch"}}, + opset_version=args.opset, + ) + print(f"exported ONNX policy to {args.out}") + + +if __name__ == "__main__": + main() +``` + +- [ ] **Step 4: Run tests to verify they pass** + +```bash +cd Training && pytest tests/test_export_onnx.py && cd .. +``` + +Expected: 3 passed. The `test_onnx_output_matches_sb3` test may occasionally show 1-2 argmax mismatches on marginal states — that's why the tolerance is `mismatches <= 2`. If it exceeds 2, something is wrong with the export. + +- [ ] **Step 5: Verify the produced ONNX file** + +```bash +ls -la models/ppo_lander.onnx +``` + +Expected: file exists, non-zero size. + +- [ ] **Step 6: Commit** + +```bash +git add Training/export_onnx.py Training/tests/test_export_onnx.py +git -c user.email=meelstorm@gmail.com -c user.name=meelstorm commit -m "feat(training): export SB3 PPO policy to ONNX with parity test" +``` + +--- + +## Task 5: End-to-end pipeline smoke + +**Files:** (no new files) + +- [ ] **Step 1: Full pipeline dry-run** + +Wipes state and re-runs the entire pipeline to prove it works end-to-end. + +```bash +rm -rf checkpoints/ models/ +dotnet publish GameCli/GameCli.csproj -c Release -o publish/GameCli +source .venv/bin/activate +cd Training +python train.py --steps 10000 --n-envs 2 --use-dummy --max-episode-steps 200 +python export_onnx.py --checkpoint ../checkpoints/ppo_lander_final.zip --out ../models/ppo_lander.onnx +cd .. +``` + +Expected: +- Training completes without error. +- `checkpoints/ppo_lander_final.zip` exists. +- `models/ppo_lander.onnx` exists and is non-empty. +- Exit code 0. + +- [ ] **Step 2: Full test suite** + +```bash +cd Training && pytest && cd .. +``` + +Expected: 11 passed, 0 failed (8 env + 3 export). + +- [ ] **Step 3: No commit — this task is verification only** + +If everything passed, Plan 2 is done. If not, fix whatever failed and iterate. + +--- + +## Definition of done for Plan 2 + +- `cd Training && pytest` reports 11 passed, 0 failed. +- Running `python Training/train.py --steps 10000 --n-envs 2 --use-dummy` produces a checkpoint under `checkpoints/`. +- Running `python Training/export_onnx.py --checkpoint checkpoints/ppo_lander_final.zip --out models/ppo_lander.onnx` produces a non-empty ONNX file. +- The exported ONNX file's argmax action matches SB3's `predict(deterministic=True)` on ≥98% of random observations. + +The next plan (Plan 3) will build the ASP.NET backend that loads this ONNX file and drives the CLI at 50 Hz per connected browser. + +--- + +## Notes on real training + +The smoke-run steps use tiny values (10k steps, 2 envs) to prove the pipeline works quickly during implementation. To actually train a useful policy, the spec calls for ~2M steps on 8 envs: + +```bash +python Training/train.py --steps 2000000 --n-envs 8 +``` + +This takes on the order of hours on a modern CPU. It's out of scope for Plan 2 (which is about building the pipeline); do it before Plan 3 needs a real model, or use a smoke-trained model to prototype the backend and swap in the real one later.