diff --git a/CLAUDE.md b/CLAUDE.md new file mode 100644 index 0000000..7215279 --- /dev/null +++ b/CLAUDE.md @@ -0,0 +1,179 @@ +# Experiment_ReinforcementLearning — cursor-following lunar lander + +A cursor-following lunar lander demo: user moves the mouse, a PPO-controlled +ship chases it against gravity. Deliberate strict separation between game +logic, backend, frontend, and training. + +Live repo: `https://gitea.tes.gd/admin/Experiment_ReinforcementLearning` + +## Architecture + +Four processes with narrow, stable interfaces: + +``` +[React UI] ──WebSocket──▶ [ASP.NET backend] ──stdin/stdout──▶ [Game CLI (.NET)] + ▲ + │ same protocol + │ + [Python PPO trainer] ──┘ +``` + +- **`GameCli/`** — C# .NET 8 console app. Stateless step-driven executable. + Reads one action per line on stdin, writes one observation JSON per line on + stdout. Owns physics, reward, observation. No networking, no UI knowledge. +- **`Backend/`** — ASP.NET Core Web API. WebSocket at `/ws/game`. Per + connection: spawns one `GameCli` subprocess, runs a 50 Hz tick loop that + calls the singleton `PolicyRunner` (ONNX) for each action. +- **`Frontend/`** — Vite + React + TypeScript. Opens a WebSocket via a + same-origin proxy (`/ws` → `localhost:5100`), sends `mousemove` as world + coords, renders the ship on a ``. +- **`Training/`** — Python. `LanderCliEnv` wraps the **same** `GameCli` binary + as a `gym.Env`, so training physics and inference physics are identical by + construction. Produces `models/ppo_lander.onnx`. + +Design spec: `docs/superpowers/specs/2026-07-17-cursor-following-lander-design.md` +Implementation plans: `docs/superpowers/plans/2026-07-17-{01-04}-*.md` + +## Build & test + +Everything runs from the repo root. + +```bash +# Game CLI (25 xUnit tests) +dotnet test --filter FullyQualifiedName~GameCli.Tests + +# Backend (8 xUnit tests — includes end-to-end WebSocket smoke) +dotnet publish GameCli/GameCli.csproj -c Release -o publish/GameCli +dotnet test --filter FullyQualifiedName~Backend.Tests + +# Training (11 pytest tests) +source .venv/bin/activate +cd Training && pytest && cd .. + +# Frontend +cd Frontend && npx tsc --noEmit && npm run build && cd .. +``` + +## Run the demo locally + +```bash +# 1. Publish CLI (idempotent) +dotnet publish GameCli/GameCli.csproj -c Release -o publish/GameCli + +# 2. Start backend (loads models/ppo_lander.onnx at startup) +ASPNETCORE_URLS=http://localhost:5100 \ + dotnet run --project Backend --no-launch-profile + +# 3. In another shell, start Vite dev server +cd Frontend && npm run dev # → http://localhost:5173/ +``` + +Open `http://localhost:5173/`. Status bar top-left shows connection state and +step counter; blue crosshair follows your mouse; white triangle is the ship. + +## Wire protocol reference + +**CLI stdio** (line-delimited JSON): + +- CLI startup: `{"init":{"world":[1,1],"dt":0.02,"obs_dim":7,"n_actions":4}}` +- Step in: `{"action":<0-3>,"target":[x,y]}` +- Reset in: `{"cmd":"reset","seed":?}` +- Step out: `{"obs":[7 floats],"state":{x,y,angle,engine},"reward":,"done":,"step":}` +- Actions: `0`=noop, `1`=left thruster, `2`=main engine, `3`=right thruster. +- Observation: `[dx, dy, vx, vy, sin(angle), cos(angle), angular_velocity]`. + +**WebSocket** (`ws://localhost:5100/ws/game` or same-origin via Vite proxy): + +- Client → server: `{"type":"cursor","x":<0-1>,"y":<0-1>}` +- Server → client (on connect): `{"type":"init","world":[1,1]}` +- Server → client (each 20 ms tick): + `{"type":"state","x":..,"y":..,"angle":..,"engine":..,"target":[..,..],"step":..}` + +Both are mirrored in code (`GameCli/Protocol.cs`, `Backend/GameProcess.cs`, +`Backend/WireMessages.cs`, `Frontend/src/protocol.ts`). + +## Coordinate & physics conventions + +- World is `[0, 1] × [0, 1]`. Y increases downward (screen-native). +- `angle = 0` = ship pointing up. Positive angle rotates clockwise. +- No walls, no ground. Ship can drift off-screen; only the reward pulls it back. +- Fixed `dt = 0.02` (50 Hz). Explicit Euler integration. +- Physics constants live in `GameCli/Physics.cs`; reward constants in + `GameCli/Reward.cs`. All `public const`, easy to tune. + +## PPO training + +```bash +source .venv/bin/activate + +# Smoke run (~30 s): +python Training/train.py --steps 10000 --n-envs 2 --use-dummy --max-episode-steps 200 + +# Real training (~hours on CPU): +python Training/train.py --steps 2000000 --n-envs 8 + +# Export to ONNX (backend reads this file): +python Training/export_onnx.py \ + --checkpoint checkpoints/ppo_lander_final.zip \ + --out models/ppo_lander.onnx +``` + +Backend loads `models/ppo_lander.onnx` at startup and fails fast if it's +missing. Restart the backend after re-exporting to pick up the new weights. + +### Current state of the model + +`models/ppo_lander.onnx` was trained for only 10k PPO steps as a +pipeline smoke — essentially random with slight bias. The ship falls off the +world in a few seconds under this policy. To see actual cursor-following +behavior, run the 2M-step training pass above (see PPO training section). + +## Config paths & env vars + +Backend resolves paths at startup by walking up from `AppContext.BaseDirectory` +to find `GameCli.sln` (the repo-root marker), then joining a relative config +path. + +- `Lander:CliPath` (default `publish/GameCli/GameCli`) +- `Lander:ModelPath` (default `models/ppo_lander.onnx`) +- Env-var overrides: `LANDER__CliPath`, `LANDER__ModelPath` (ASP.NET + double-underscore convention). + +## Repo conventions + +- Every commit lands on `main` (no PR flow — greenfield project). +- Author: `meelstorm `. Use + `-c user.email=meelstorm@gmail.com -c user.name=meelstorm` when committing. +- Conventional commit prefixes: `feat(gamecli):`, `feat(backend):`, + `feat(training):`, `feat(frontend):`, `fix(...)`, `test(...)`, `chore:`. +- `.gitignore` covers `bin/`, `obj/`, `node_modules/`, `dist/`, `publish/`, + `.venv/`, `checkpoints/`, `models/`, `tensorboard/`, `*.onnx`. +- The trained model is NOT in git — it's a build artifact. + +## Known limitations & future work + +- **Model is undertrained** — see "Current state of the model" above. +- **No walls** — a badly-trained policy loses to gravity within a second and + the ship drifts off-screen forever. Design choice: reward should be enough + to keep a well-trained policy in-world. +- **Angle wrap in reward is applied** but not in `Physics.Step` — the ship's + `angle` field grows unbounded across many revolutions. Observation uses + sin/cos so it's immune; reward wraps to `[-π, π]` internally. +- **StrictMode WebSocket double-mount** in dev — React's StrictMode mounts + the `useGameSocket` effect twice, so you'll see one benign + "closed before established" WebSocket warning per page load. +- **Single-player only** — one CLI subprocess per connection; multi-user + demo is fine at demo scale but there's no shared world. + +## Debugging tips + +- **Backend can't find CLI or ONNX at startup** — the repo-root walker looks + for `GameCli.sln`. Both `.sln` and `.slnx` formats work. +- **`dotnet test` in `Backend.Tests` uses the CLI** — publish first + (`dotnet publish GameCli -c Release -o publish/GameCli`) so + `EndToEndSmokeTests` can find the binary. +- **`torch.onnx.export` on torch 2.13+** — pass `dynamo=False` to force the + legacy TorchScript exporter unless `onnxscript` is installed. +- **Vite WS proxy failing** — verify with a Node script bypassing the proxy + (`ws://localhost:5100/ws/game` direct). If direct works but proxy doesn't, + check `Frontend/vite.config.ts` proxy block.