Add Plan 2: Python PPO training pipeline plan

2026-07-17 17:02:15 +00:00
parent d1290442ed
commit 30ac7a8114
1 changed files with 789 additions and 0 deletions
--- a/docs/superpowers/plans/2026-07-17-02-training-pipeline.md
+++ b/docs/superpowers/plans/2026-07-17-02-training-pipeline.md
@@ -0,0 +1,789 @@
+# Training Pipeline Implementation Plan
+
+> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking.
+
+**Goal:** Build the Python side of the design: a `LanderCliEnv` Gymnasium environment that wraps the `GameCli` binary as a subprocess, a `train.py` that trains a PPO policy on N vectorized copies of it, and an `export_onnx.py` that exports the trained network as `models/ppo_lander.onnx` for the ASP.NET backend to load. Reference spec at `docs/superpowers/specs/2026-07-17-cursor-following-lander-design.md`.
+
+**Architecture:** All Python code lives under `Training/`. `LanderCliEnv` speaks the exact protocol the `GameCli` documents on stdio, so training uses literally the same physics as inference. Training runs a `SubprocVecEnv` of N envs → PPO batches transitions → checkpoints to `checkpoints/` → export → verify → `models/ppo_lander.onnx`.
+
+**Tech Stack:** Python 3.11 or 3.12, `gymnasium`, `stable-baselines3`, `torch`, `onnx`, `onnxruntime`, `pytest`.
+
+---
+
+## Prerequisites
+
+- **Plan 1 complete.** The `GameCli` binary must be publishable at `publish/GameCli/GameCli` (run `dotnet publish GameCli/GameCli.csproj -c Release -o publish/GameCli` if it's not there yet). All Training/ code assumes that path.
+- Python 3.11 or 3.12 available as `python3`.
+
+---
+
+## File structure
+
+```
+Experiment_ReinforcementLearning/
+├── Training/
+│   ├── requirements.txt
+│   ├── pytest.ini
+│   ├── lander_cli_env.py          # LanderCliEnv(gym.Env)
+│   ├── train.py                   # PPO training with SubprocVecEnv
+│   ├── export_onnx.py             # SB3 -> ONNX
+│   └── tests/
+│       ├── __init__.py
+│       ├── test_lander_cli_env.py
+│       └── test_export_onnx.py
+├── models/                        # gitignored; export_onnx.py writes ppo_lander.onnx here
+├── checkpoints/                   # gitignored; SB3 checkpoints
+└── tensorboard/                   # gitignored; TB logs
+```
+
+`Training/lander_cli_env.py` is the single file that touches the subprocess. `train.py` and `export_onnx.py` just consume Gymnasium/SB3 APIs.
+
+---
+
+## Task 1: Scaffold Python project
+
+**Files:**
+- Create: `Training/requirements.txt`
+- Create: `Training/pytest.ini`
+- Create: `Training/tests/__init__.py`
+
+- [ ] **Step 1: Check Python is available**
+
+Run: `python3 --version`
+Expected: `Python 3.11.x` or `3.12.x`. If older or missing, STOP and report BLOCKED.
+
+- [ ] **Step 2: Create the virtualenv at repo root**
+
+```bash
+python3 -m venv .venv
+source .venv/bin/activate
+pip install --upgrade pip
+```
+
+- [ ] **Step 3: Write `Training/requirements.txt`**
+
+```
+gymnasium>=0.29,<2
+stable-baselines3>=2.3,<3
+torch>=2.2,<3
+onnx>=1.16,<2
+onnxruntime>=1.18,<2
+numpy>=1.24,<3
+tensorboard>=2.15
+pytest>=8
+```
+
+- [ ] **Step 4: Install dependencies**
+
+Run: `pip install -r Training/requirements.txt`
+Expected: successful install. This may take a few minutes (torch is large).
+
+- [ ] **Step 5: Write `Training/pytest.ini`**
+
+```ini
+[pytest]
+testpaths = tests
+python_files = test_*.py
+addopts = -v
+```
+
+- [ ] **Step 6: Write `Training/tests/__init__.py`** (empty file)
+
+```bash
+touch Training/tests/__init__.py
+```
+
+- [ ] **Step 7: Verify pytest can discover an empty test suite**
+
+From `Training/` directory:
+
+```bash
+cd Training && pytest && cd ..
+```
+
+Expected: `no tests ran` (0 tests, exit code 0 or 5 — both mean "no tests but no failures"). Exit code 5 is fine.
+
+- [ ] **Step 8: Commit**
+
+```bash
+git add Training/
+git -c user.email=meelstorm@gmail.com -c user.name=meelstorm commit -m "feat(training): scaffold Python training project"
+```
+
+---
+
+## Task 2: `LanderCliEnv` — Gymnasium environment wrapping the CLI
+
+**Files:**
+- Create: `Training/lander_cli_env.py`
+- Create: `Training/tests/test_lander_cli_env.py`
+
+### Protocol quick reference (from Plan 1)
+
+- CLI startup emits: `{"init":{"world":[1,1],"dt":0.02,"obs_dim":7,"n_actions":4}}`
+- Step input: `{"action":<int>,"target":[<float>,<float>]}`
+- Reset input: `{"cmd":"reset","seed":<int>}` (seed optional)
+- Output line: `{"obs":[...7 floats...],"state":{...},"reward":<float>,"done":<bool>,"step":<int>}`
+- CLI binary path (relative to repo root): `publish/GameCli/GameCli`
+
+- [ ] **Step 1: Write the failing tests**
+
+Create `Training/tests/test_lander_cli_env.py`:
+
+```python
+"""Unit tests for LanderCliEnv."""
+from pathlib import Path
+
+import numpy as np
+import pytest
+from gymnasium.utils.env_checker import check_env
+
+# Add parent dir to sys.path so we can import lander_cli_env
+import sys
+sys.path.insert(0, str(Path(__file__).resolve().parent.parent))
+
+from lander_cli_env import LanderCliEnv
+
+
+REPO_ROOT = Path(__file__).resolve().parents[2]
+CLI_BINARY = REPO_ROOT / "publish" / "GameCli" / "GameCli"
+
+
+@pytest.fixture(scope="module")
+def cli_binary_exists():
+    if not CLI_BINARY.exists():
+        pytest.skip(f"CLI binary not found at {CLI_BINARY} — run `dotnet publish GameCli -c Release -o publish/GameCli`")
+    return CLI_BINARY
+
+
+def test_env_opens_and_closes(cli_binary_exists):
+    env = LanderCliEnv(cli_binary=str(cli_binary_exists))
+    env.close()  # should not raise
+
+
+def test_observation_space_is_seven_dim(cli_binary_exists):
+    env = LanderCliEnv(cli_binary=str(cli_binary_exists))
+    try:
+        assert env.observation_space.shape == (7,)
+    finally:
+        env.close()
+
+
+def test_action_space_is_discrete_four(cli_binary_exists):
+    env = LanderCliEnv(cli_binary=str(cli_binary_exists))
+    try:
+        assert env.action_space.n == 4
+    finally:
+        env.close()
+
+
+def test_reset_returns_seven_dim_observation(cli_binary_exists):
+    env = LanderCliEnv(cli_binary=str(cli_binary_exists))
+    try:
+        obs, info = env.reset(seed=42)
+        assert isinstance(obs, np.ndarray)
+        assert obs.shape == (7,)
+        assert obs.dtype == np.float32
+    finally:
+        env.close()
+
+
+def test_step_advances_and_returns_five_tuple(cli_binary_exists):
+    env = LanderCliEnv(cli_binary=str(cli_binary_exists))
+    try:
+        env.reset(seed=0)
+        obs, reward, terminated, truncated, info = env.step(0)  # noop
+        assert obs.shape == (7,)
+        assert isinstance(reward, float)
+        assert terminated is False  # never terminates
+        assert isinstance(truncated, bool)
+    finally:
+        env.close()
+
+
+def test_reset_randomizes_target_across_episodes(cli_binary_exists):
+    env = LanderCliEnv(cli_binary=str(cli_binary_exists))
+    try:
+        # Two resets with different seeds should generally give different targets.
+        env.reset(seed=1)
+        target1 = env._target
+        env.reset(seed=2)
+        target2 = env._target
+        assert target1 != target2
+    finally:
+        env.close()
+
+
+def test_truncation_at_max_steps(cli_binary_exists):
+    env = LanderCliEnv(cli_binary=str(cli_binary_exists), max_episode_steps=5)
+    try:
+        env.reset(seed=0)
+        truncated = False
+        for _ in range(5):
+            _, _, _, truncated, _ = env.step(0)
+        assert truncated is True
+    finally:
+        env.close()
+
+
+def test_check_env_passes(cli_binary_exists):
+    """Gymnasium's own env sanity checker — spaces, dtypes, reset/step contract."""
+    env = LanderCliEnv(cli_binary=str(cli_binary_exists), max_episode_steps=50)
+    try:
+        # skip_render_check because we don't implement rendering
+        check_env(env.unwrapped, skip_render_check=True)
+    finally:
+        env.close()
+```
+
+- [ ] **Step 2: Run tests to verify they fail**
+
+```bash
+cd Training && pytest && cd ..
+```
+
+Expected: import error — `lander_cli_env` module does not exist.
+
+- [ ] **Step 3: Write `Training/lander_cli_env.py`**
+
+```python
+"""Gymnasium environment that wraps the GameCli C# binary as a subprocess.
+
+Speaks the line-delimited JSON protocol defined in:
+  docs/superpowers/specs/2026-07-17-cursor-following-lander-design.md
+
+Each episode picks a random target in [0,1]^2 at reset time and holds it fixed.
+Never terminates — episodes end via truncation at max_episode_steps.
+"""
+from __future__ import annotations
+
+import json
+import subprocess
+from pathlib import Path
+from typing import Any, Optional
+
+import gymnasium as gym
+import numpy as np
+from gymnasium import spaces
+
+
+class LanderCliEnv(gym.Env):
+    """One CLI subprocess per env instance. Not thread-safe; safe under SubprocVecEnv."""
+
+    metadata = {"render_modes": []}
+
+    def __init__(
+        self,
+        cli_binary: str,
+        max_episode_steps: int = 1000,
+    ) -> None:
+        super().__init__()
+        self._cli_binary = cli_binary
+        self._max_episode_steps = max_episode_steps
+
+        self._proc: Optional[subprocess.Popen] = None
+        self._target: tuple[float, float] = (0.5, 0.5)
+        self._step_count: int = 0
+
+        self._start_process()
+        self._read_init()
+
+        self.observation_space = spaces.Box(
+            low=-np.inf, high=np.inf, shape=(7,), dtype=np.float32
+        )
+        self.action_space = spaces.Discrete(4)
+
+    # -------- subprocess lifecycle --------
+
+    def _start_process(self) -> None:
+        self._proc = subprocess.Popen(
+            [self._cli_binary],
+            stdin=subprocess.PIPE,
+            stdout=subprocess.PIPE,
+            stderr=subprocess.PIPE,
+            text=True,
+            bufsize=1,  # line-buffered
+        )
+
+    def _read_init(self) -> None:
+        assert self._proc is not None and self._proc.stdout is not None
+        line = self._proc.stdout.readline()
+        if not line:
+            raise RuntimeError("CLI died before emitting init handshake")
+        msg = json.loads(line)
+        if "init" not in msg:
+            raise RuntimeError(f"expected init handshake, got: {line!r}")
+
+    def _send(self, obj: dict[str, Any]) -> None:
+        assert self._proc is not None and self._proc.stdin is not None
+        self._proc.stdin.write(json.dumps(obj) + "\n")
+        self._proc.stdin.flush()
+
+    def _recv(self) -> dict[str, Any]:
+        assert self._proc is not None and self._proc.stdout is not None
+        line = self._proc.stdout.readline()
+        if not line:
+            raise RuntimeError("CLI closed stdout unexpectedly")
+        return json.loads(line)
+
+    # -------- gym.Env API --------
+
+    def reset(
+        self, *, seed: Optional[int] = None, options: Optional[dict] = None
+    ) -> tuple[np.ndarray, dict]:
+        super().reset(seed=seed)
+        # Sample a fresh target each episode; the Gymnasium-provided
+        # np_random is deterministic given `seed`.
+        tx = float(self.np_random.uniform(0.05, 0.95))
+        ty = float(self.np_random.uniform(0.05, 0.95))
+        self._target = (tx, ty)
+        self._step_count = 0
+
+        cmd: dict[str, Any] = {"cmd": "reset"}
+        if seed is not None:
+            cmd["seed"] = int(seed)
+        self._send(cmd)
+        msg = self._recv()
+        obs = np.asarray(msg["obs"], dtype=np.float32)
+        # Overwrite the CLI's dummy (0.5, 0.5) target displacement with the real one.
+        # dx = target.X - ship.X, dy = target.Y - ship.Y.
+        obs[0] = tx - msg["state"]["x"]
+        obs[1] = ty - msg["state"]["y"]
+        return obs, {}
+
+    def step(self, action: int) -> tuple[np.ndarray, float, bool, bool, dict]:
+        self._send({"action": int(action), "target": list(self._target)})
+        msg = self._recv()
+        obs = np.asarray(msg["obs"], dtype=np.float32)
+        reward = float(msg["reward"])
+        self._step_count += 1
+        truncated = self._step_count >= self._max_episode_steps
+        terminated = False  # continuous hover task
+        return obs, reward, terminated, truncated, {}
+
+    def close(self) -> None:
+        if self._proc is None:
+            return
+        try:
+            if self._proc.stdin is not None:
+                self._proc.stdin.close()
+        except (BrokenPipeError, OSError):
+            pass
+        try:
+            self._proc.wait(timeout=3)
+        except subprocess.TimeoutExpired:
+            self._proc.kill()
+            self._proc.wait(timeout=1)
+        self._proc = None
+
+    def __del__(self) -> None:
+        try:
+            self.close()
+        except Exception:
+            pass
+```
+
+- [ ] **Step 4: Publish the CLI binary (prerequisite for tests)**
+
+If `publish/GameCli/GameCli` doesn't exist yet, run:
+
+```bash
+dotnet publish GameCli/GameCli.csproj -c Release -o publish/GameCli
+```
+
+- [ ] **Step 5: Run tests to verify they pass**
+
+```bash
+cd Training && pytest tests/test_lander_cli_env.py && cd ..
+```
+
+Expected: 8 passed, 0 failed.
+
+- [ ] **Step 6: Commit**
+
+```bash
+git add Training/lander_cli_env.py Training/tests/test_lander_cli_env.py
+git -c user.email=meelstorm@gmail.com -c user.name=meelstorm commit -m "feat(training): implement LanderCliEnv wrapping the GameCli subprocess"
+```
+
+---
+
+## Task 3: `train.py` — PPO training with vectorized envs
+
+**Files:**
+- Create: `Training/train.py`
+
+- [ ] **Step 1: Write `Training/train.py`**
+
+```python
+"""Train a PPO policy on LanderCliEnv.
+
+Usage:
+    python train.py --steps 100000                 # short smoke run
+    python train.py --steps 2000000 --n-envs 8     # full training
+"""
+from __future__ import annotations
+
+import argparse
+from pathlib import Path
+
+from stable_baselines3 import PPO
+from stable_baselines3.common.callbacks import CheckpointCallback
+from stable_baselines3.common.vec_env import SubprocVecEnv, DummyVecEnv
+
+# sys.path shim so this script can be run from any CWD.
+import sys
+sys.path.insert(0, str(Path(__file__).resolve().parent))
+
+from lander_cli_env import LanderCliEnv
+
+
+REPO_ROOT = Path(__file__).resolve().parents[1]
+CLI_BINARY = REPO_ROOT / "publish" / "GameCli" / "GameCli"
+
+
+def make_env(seed: int, max_episode_steps: int):
+    """Return a thunk that SubprocVecEnv can call to construct one env."""
+    def _fn():
+        env = LanderCliEnv(
+            cli_binary=str(CLI_BINARY),
+            max_episode_steps=max_episode_steps,
+        )
+        env.reset(seed=seed)
+        return env
+    return _fn
+
+
+def main() -> None:
+    parser = argparse.ArgumentParser()
+    parser.add_argument("--steps", type=int, default=2_000_000,
+                        help="total env steps to train for")
+    parser.add_argument("--n-envs", type=int, default=8,
+                        help="parallel envs; each spawns one GameCli subprocess")
+    parser.add_argument("--max-episode-steps", type=int, default=1000)
+    parser.add_argument("--seed", type=int, default=0)
+    parser.add_argument("--checkpoint-dir", type=Path,
+                        default=REPO_ROOT / "checkpoints")
+    parser.add_argument("--tb-dir", type=Path,
+                        default=REPO_ROOT / "tensorboard")
+    parser.add_argument("--use-dummy", action="store_true",
+                        help="run envs in-process (slower, easier to debug)")
+    args = parser.parse_args()
+
+    if not CLI_BINARY.exists():
+        raise SystemExit(
+            f"CLI binary not found at {CLI_BINARY}. Run: "
+            f"dotnet publish GameCli/GameCli.csproj -c Release -o publish/GameCli"
+        )
+
+    args.checkpoint_dir.mkdir(parents=True, exist_ok=True)
+    args.tb_dir.mkdir(parents=True, exist_ok=True)
+
+    env_fns = [
+        make_env(seed=args.seed + i, max_episode_steps=args.max_episode_steps)
+        for i in range(args.n_envs)
+    ]
+    vec_env_cls = DummyVecEnv if args.use_dummy else SubprocVecEnv
+    vec_env = vec_env_cls(env_fns)
+
+    model = PPO(
+        "MlpPolicy",
+        vec_env,
+        verbose=1,
+        seed=args.seed,
+        tensorboard_log=str(args.tb_dir),
+        policy_kwargs=dict(net_arch=[64, 64]),
+    )
+
+    checkpoint_cb = CheckpointCallback(
+        save_freq=max(args.steps // 10, 1) // args.n_envs,
+        save_path=str(args.checkpoint_dir),
+        name_prefix="ppo_lander",
+    )
+
+    try:
+        model.learn(total_timesteps=args.steps, callback=checkpoint_cb)
+        final_path = args.checkpoint_dir / "ppo_lander_final.zip"
+        model.save(str(final_path))
+        print(f"saved final model to {final_path}")
+    finally:
+        vec_env.close()
+
+
+if __name__ == "__main__":
+    main()
+```
+
+- [ ] **Step 2: Publish binary (idempotent)**
+
+```bash
+dotnet publish GameCli/GameCli.csproj -c Release -o publish/GameCli
+```
+
+- [ ] **Step 3: Smoke-run training for a small step count**
+
+```bash
+source .venv/bin/activate
+cd Training && python train.py --steps 5000 --n-envs 2 --use-dummy --max-episode-steps 200 && cd ..
+```
+
+Expected:
+- SB3 prints the standard rollout table (`| rollout/ | ...`).
+- Final line: `saved final model to .../checkpoints/ppo_lander_final.zip`.
+- Exit code 0.
+- The file `checkpoints/ppo_lander_final.zip` exists.
+
+The `--use-dummy` flag runs envs in-process which is easier for debugging. Real training uses SubprocVecEnv (leave that flag off).
+
+- [ ] **Step 4: Commit**
+
+```bash
+git add Training/train.py
+git -c user.email=meelstorm@gmail.com -c user.name=meelstorm commit -m "feat(training): PPO training script with vectorized LanderCliEnv"
+```
+
+---
+
+## Task 4: `export_onnx.py` — export trained policy to ONNX
+
+**Files:**
+- Create: `Training/export_onnx.py`
+- Create: `Training/tests/test_export_onnx.py`
+
+- [ ] **Step 1: Write the failing tests**
+
+Create `Training/tests/test_export_onnx.py`:
+
+```python
+"""Verify export_onnx.py produces an ONNX file that matches the SB3 policy."""
+from pathlib import Path
+
+import numpy as np
+import pytest
+
+
+REPO_ROOT = Path(__file__).resolve().parents[2]
+CHECKPOINT = REPO_ROOT / "checkpoints" / "ppo_lander_final.zip"
+ONNX_OUT = REPO_ROOT / "models" / "ppo_lander.onnx"
+
+
+@pytest.fixture(scope="module")
+def exported_onnx():
+    if not CHECKPOINT.exists():
+        pytest.skip(f"no checkpoint at {CHECKPOINT} — run train.py first")
+    # export
+    import subprocess
+    import sys
+    training_dir = REPO_ROOT / "Training"
+    result = subprocess.run(
+        [sys.executable, str(training_dir / "export_onnx.py"),
+         "--checkpoint", str(CHECKPOINT), "--out", str(ONNX_OUT)],
+        capture_output=True, text=True, cwd=str(REPO_ROOT),
+    )
+    assert result.returncode == 0, f"export failed: {result.stderr}"
+    assert ONNX_OUT.exists(), f"ONNX file not produced at {ONNX_OUT}"
+    return ONNX_OUT
+
+
+def test_onnx_file_exists(exported_onnx):
+    assert exported_onnx.stat().st_size > 0
+
+
+def test_onnx_input_output_shapes(exported_onnx):
+    import onnx
+    model = onnx.load(str(exported_onnx))
+    assert len(model.graph.input) == 1
+    in_shape = [d.dim_value for d in model.graph.input[0].type.tensor_type.shape.dim]
+    # (batch, 7)
+    assert in_shape[-1] == 7
+    assert len(model.graph.output) >= 1
+    out_shape = [d.dim_value for d in model.graph.output[0].type.tensor_type.shape.dim]
+    assert out_shape[-1] == 4  # 4 discrete actions
+
+
+def test_onnx_output_matches_sb3(exported_onnx):
+    """Sanity: ONNX inference for 100 random obs must argmax to the same action as SB3."""
+    import onnxruntime
+    from stable_baselines3 import PPO
+
+    model = PPO.load(str(CHECKPOINT))
+    session = onnxruntime.InferenceSession(str(exported_onnx))
+    input_name = session.get_inputs()[0].name
+
+    rng = np.random.default_rng(1234)
+    obs_batch = rng.normal(size=(100, 7)).astype(np.float32)
+
+    # ONNX: (100, 4) logits
+    onnx_logits = session.run(None, {input_name: obs_batch})[0]
+    onnx_actions = onnx_logits.argmax(axis=-1)
+
+    # SB3 deterministic prediction
+    sb3_actions, _ = model.predict(obs_batch, deterministic=True)
+
+    mismatches = int((onnx_actions != sb3_actions).sum())
+    # Allow up to 2 mismatches out of 100 for float rounding on marginal states.
+    assert mismatches <= 2, f"{mismatches}/100 argmax mismatches between ONNX and SB3"
+```
+
+- [ ] **Step 2: Run tests to verify they fail**
+
+```bash
+cd Training && pytest tests/test_export_onnx.py && cd ..
+```
+
+Expected: fail because `export_onnx.py` does not exist.
+
+- [ ] **Step 3: Write `Training/export_onnx.py`**
+
+```python
+"""Export a trained SB3 PPO policy to ONNX for the .NET backend.
+
+The exported model has a single float input of shape [batch, 7] (the observation
+vector) and a single float output of shape [batch, 4] (action logits). Argmax
+over the last axis gives the action.
+"""
+from __future__ import annotations
+
+import argparse
+from pathlib import Path
+
+import torch
+from stable_baselines3 import PPO
+
+
+class OnnxablePolicy(torch.nn.Module):
+    """Wraps the SB3 policy's actor path for ONNX export.
+
+    Standard SB3 MlpPolicy for a discrete action space:
+        features = policy.extract_features(obs)
+        latent_pi = policy.mlp_extractor.forward_actor(features)
+        logits = policy.action_net(latent_pi)
+    """
+
+    def __init__(self, policy: torch.nn.Module) -> None:
+        super().__init__()
+        self.policy = policy
+
+    def forward(self, obs: torch.Tensor) -> torch.Tensor:
+        features = self.policy.extract_features(obs)
+        latent_pi = self.policy.mlp_extractor.forward_actor(features)
+        return self.policy.action_net(latent_pi)
+
+
+def main() -> None:
+    parser = argparse.ArgumentParser()
+    parser.add_argument("--checkpoint", type=Path, required=True,
+                        help="path to SB3 .zip checkpoint")
+    parser.add_argument("--out", type=Path, required=True,
+                        help="destination .onnx path")
+    parser.add_argument("--opset", type=int, default=17)
+    args = parser.parse_args()
+
+    args.out.parent.mkdir(parents=True, exist_ok=True)
+
+    model = PPO.load(str(args.checkpoint), device="cpu")
+    model.policy.eval()
+
+    onnxable = OnnxablePolicy(model.policy)
+    onnxable.eval()
+
+    dummy = torch.randn(1, 7, dtype=torch.float32)
+    torch.onnx.export(
+        onnxable,
+        dummy,
+        str(args.out),
+        input_names=["obs"],
+        output_names=["logits"],
+        dynamic_axes={"obs": {0: "batch"}, "logits": {0: "batch"}},
+        opset_version=args.opset,
+    )
+    print(f"exported ONNX policy to {args.out}")
+
+
+if __name__ == "__main__":
+    main()
+```
+
+- [ ] **Step 4: Run tests to verify they pass**
+
+```bash
+cd Training && pytest tests/test_export_onnx.py && cd ..
+```
+
+Expected: 3 passed. The `test_onnx_output_matches_sb3` test may occasionally show 1-2 argmax mismatches on marginal states — that's why the tolerance is `mismatches <= 2`. If it exceeds 2, something is wrong with the export.
+
+- [ ] **Step 5: Verify the produced ONNX file**
+
+```bash
+ls -la models/ppo_lander.onnx
+```
+
+Expected: file exists, non-zero size.
+
+- [ ] **Step 6: Commit**
+
+```bash
+git add Training/export_onnx.py Training/tests/test_export_onnx.py
+git -c user.email=meelstorm@gmail.com -c user.name=meelstorm commit -m "feat(training): export SB3 PPO policy to ONNX with parity test"
+```
+
+---
+
+## Task 5: End-to-end pipeline smoke
+
+**Files:** (no new files)
+
+- [ ] **Step 1: Full pipeline dry-run**
+
+Wipes state and re-runs the entire pipeline to prove it works end-to-end.
+
+```bash
+rm -rf checkpoints/ models/
+dotnet publish GameCli/GameCli.csproj -c Release -o publish/GameCli
+source .venv/bin/activate
+cd Training
+python train.py --steps 10000 --n-envs 2 --use-dummy --max-episode-steps 200
+python export_onnx.py --checkpoint ../checkpoints/ppo_lander_final.zip --out ../models/ppo_lander.onnx
+cd ..
+```
+
+Expected:
+- Training completes without error.
+- `checkpoints/ppo_lander_final.zip` exists.
+- `models/ppo_lander.onnx` exists and is non-empty.
+- Exit code 0.
+
+- [ ] **Step 2: Full test suite**
+
+```bash
+cd Training && pytest && cd ..
+```
+
+Expected: 11 passed, 0 failed (8 env + 3 export).
+
+- [ ] **Step 3: No commit — this task is verification only**
+
+If everything passed, Plan 2 is done. If not, fix whatever failed and iterate.
+
+---
+
+## Definition of done for Plan 2
+
+- `cd Training && pytest` reports 11 passed, 0 failed.
+- Running `python Training/train.py --steps 10000 --n-envs 2 --use-dummy` produces a checkpoint under `checkpoints/`.
+- Running `python Training/export_onnx.py --checkpoint checkpoints/ppo_lander_final.zip --out models/ppo_lander.onnx` produces a non-empty ONNX file.
+- The exported ONNX file's argmax action matches SB3's `predict(deterministic=True)` on ≥98% of random observations.
+
+The next plan (Plan 3) will build the ASP.NET backend that loads this ONNX file and drives the CLI at 50 Hz per connected browser.
+
+---
+
+## Notes on real training
+
+The smoke-run steps use tiny values (10k steps, 2 envs) to prove the pipeline works quickly during implementation. To actually train a useful policy, the spec calls for ~2M steps on 8 envs:
+
+```bash
+python Training/train.py --steps 2000000 --n-envs 8
+```
+
+This takes on the order of hours on a modern CPU. It's out of scope for Plan 2 (which is about building the pipeline); do it before Plan 3 needs a real model, or use a smoke-trained model to prototype the backend and swap in the real one later.