180 lines
7.5 KiB
Markdown
180 lines
7.5 KiB
Markdown
# Experiment_ReinforcementLearning — cursor-following lunar lander
|
||
|
||
A cursor-following lunar lander demo: user moves the mouse, a PPO-controlled
|
||
ship chases it against gravity. Deliberate strict separation between game
|
||
logic, backend, frontend, and training.
|
||
|
||
Live repo: `https://gitea.tes.gd/admin/Experiment_ReinforcementLearning`
|
||
|
||
## Architecture
|
||
|
||
Four processes with narrow, stable interfaces:
|
||
|
||
```
|
||
[React UI] ──WebSocket──▶ [ASP.NET backend] ──stdin/stdout──▶ [Game CLI (.NET)]
|
||
▲
|
||
│ same protocol
|
||
│
|
||
[Python PPO trainer] ──┘
|
||
```
|
||
|
||
- **`GameCli/`** — C# .NET 8 console app. Stateless step-driven executable.
|
||
Reads one action per line on stdin, writes one observation JSON per line on
|
||
stdout. Owns physics, reward, observation. No networking, no UI knowledge.
|
||
- **`Backend/`** — ASP.NET Core Web API. WebSocket at `/ws/game`. Per
|
||
connection: spawns one `GameCli` subprocess, runs a 50 Hz tick loop that
|
||
calls the singleton `PolicyRunner` (ONNX) for each action.
|
||
- **`Frontend/`** — Vite + React + TypeScript. Opens a WebSocket via a
|
||
same-origin proxy (`/ws` → `localhost:5100`), sends `mousemove` as world
|
||
coords, renders the ship on a `<canvas>`.
|
||
- **`Training/`** — Python. `LanderCliEnv` wraps the **same** `GameCli` binary
|
||
as a `gym.Env`, so training physics and inference physics are identical by
|
||
construction. Produces `models/ppo_lander.onnx`.
|
||
|
||
Design spec: `docs/superpowers/specs/2026-07-17-cursor-following-lander-design.md`
|
||
Implementation plans: `docs/superpowers/plans/2026-07-17-{01-04}-*.md`
|
||
|
||
## Build & test
|
||
|
||
Everything runs from the repo root.
|
||
|
||
```bash
|
||
# Game CLI (25 xUnit tests)
|
||
dotnet test --filter FullyQualifiedName~GameCli.Tests
|
||
|
||
# Backend (8 xUnit tests — includes end-to-end WebSocket smoke)
|
||
dotnet publish GameCli/GameCli.csproj -c Release -o publish/GameCli
|
||
dotnet test --filter FullyQualifiedName~Backend.Tests
|
||
|
||
# Training (11 pytest tests)
|
||
source .venv/bin/activate
|
||
cd Training && pytest && cd ..
|
||
|
||
# Frontend
|
||
cd Frontend && npx tsc --noEmit && npm run build && cd ..
|
||
```
|
||
|
||
## Run the demo locally
|
||
|
||
```bash
|
||
# 1. Publish CLI (idempotent)
|
||
dotnet publish GameCli/GameCli.csproj -c Release -o publish/GameCli
|
||
|
||
# 2. Start backend (loads models/ppo_lander.onnx at startup)
|
||
ASPNETCORE_URLS=http://localhost:5100 \
|
||
dotnet run --project Backend --no-launch-profile
|
||
|
||
# 3. In another shell, start Vite dev server
|
||
cd Frontend && npm run dev # → http://localhost:5173/
|
||
```
|
||
|
||
Open `http://localhost:5173/`. Status bar top-left shows connection state and
|
||
step counter; blue crosshair follows your mouse; white triangle is the ship.
|
||
|
||
## Wire protocol reference
|
||
|
||
**CLI stdio** (line-delimited JSON):
|
||
|
||
- CLI startup: `{"init":{"world":[1,1],"dt":0.02,"obs_dim":7,"n_actions":4}}`
|
||
- Step in: `{"action":<0-3>,"target":[x,y]}`
|
||
- Reset in: `{"cmd":"reset","seed":<int>?}`
|
||
- Step out: `{"obs":[7 floats],"state":{x,y,angle,engine},"reward":<f>,"done":<b>,"step":<i>}`
|
||
- Actions: `0`=noop, `1`=left thruster, `2`=main engine, `3`=right thruster.
|
||
- Observation: `[dx, dy, vx, vy, sin(angle), cos(angle), angular_velocity]`.
|
||
|
||
**WebSocket** (`ws://localhost:5100/ws/game` or same-origin via Vite proxy):
|
||
|
||
- Client → server: `{"type":"cursor","x":<0-1>,"y":<0-1>}`
|
||
- Server → client (on connect): `{"type":"init","world":[1,1]}`
|
||
- Server → client (each 20 ms tick):
|
||
`{"type":"state","x":..,"y":..,"angle":..,"engine":..,"target":[..,..],"step":..}`
|
||
|
||
Both are mirrored in code (`GameCli/Protocol.cs`, `Backend/GameProcess.cs`,
|
||
`Backend/WireMessages.cs`, `Frontend/src/protocol.ts`).
|
||
|
||
## Coordinate & physics conventions
|
||
|
||
- World is `[0, 1] × [0, 1]`. Y increases downward (screen-native).
|
||
- `angle = 0` = ship pointing up. Positive angle rotates clockwise.
|
||
- No walls, no ground. Ship can drift off-screen; only the reward pulls it back.
|
||
- Fixed `dt = 0.02` (50 Hz). Explicit Euler integration.
|
||
- Physics constants live in `GameCli/Physics.cs`; reward constants in
|
||
`GameCli/Reward.cs`. All `public const`, easy to tune.
|
||
|
||
## PPO training
|
||
|
||
```bash
|
||
source .venv/bin/activate
|
||
|
||
# Smoke run (~30 s):
|
||
python Training/train.py --steps 10000 --n-envs 2 --use-dummy --max-episode-steps 200
|
||
|
||
# Real training (~hours on CPU):
|
||
python Training/train.py --steps 2000000 --n-envs 8
|
||
|
||
# Export to ONNX (backend reads this file):
|
||
python Training/export_onnx.py \
|
||
--checkpoint checkpoints/ppo_lander_final.zip \
|
||
--out models/ppo_lander.onnx
|
||
```
|
||
|
||
Backend loads `models/ppo_lander.onnx` at startup and fails fast if it's
|
||
missing. Restart the backend after re-exporting to pick up the new weights.
|
||
|
||
### Current state of the model
|
||
|
||
`models/ppo_lander.onnx` was trained for only 10k PPO steps as a
|
||
pipeline smoke — essentially random with slight bias. The ship falls off the
|
||
world in a few seconds under this policy. To see actual cursor-following
|
||
behavior, run the 2M-step training pass above (see PPO training section).
|
||
|
||
## Config paths & env vars
|
||
|
||
Backend resolves paths at startup by walking up from `AppContext.BaseDirectory`
|
||
to find `GameCli.sln` (the repo-root marker), then joining a relative config
|
||
path.
|
||
|
||
- `Lander:CliPath` (default `publish/GameCli/GameCli`)
|
||
- `Lander:ModelPath` (default `models/ppo_lander.onnx`)
|
||
- Env-var overrides: `LANDER__CliPath`, `LANDER__ModelPath` (ASP.NET
|
||
double-underscore convention).
|
||
|
||
## Repo conventions
|
||
|
||
- Every commit lands on `main` (no PR flow — greenfield project).
|
||
- Author: `meelstorm <meelstorm@gmail.com>`. Use
|
||
`-c user.email=meelstorm@gmail.com -c user.name=meelstorm` when committing.
|
||
- Conventional commit prefixes: `feat(gamecli):`, `feat(backend):`,
|
||
`feat(training):`, `feat(frontend):`, `fix(...)`, `test(...)`, `chore:`.
|
||
- `.gitignore` covers `bin/`, `obj/`, `node_modules/`, `dist/`, `publish/`,
|
||
`.venv/`, `checkpoints/`, `models/`, `tensorboard/`, `*.onnx`.
|
||
- The trained model is NOT in git — it's a build artifact.
|
||
|
||
## Known limitations & future work
|
||
|
||
- **Model is undertrained** — see "Current state of the model" above.
|
||
- **No walls** — a badly-trained policy loses to gravity within a second and
|
||
the ship drifts off-screen forever. Design choice: reward should be enough
|
||
to keep a well-trained policy in-world.
|
||
- **Angle wrap in reward is applied** but not in `Physics.Step` — the ship's
|
||
`angle` field grows unbounded across many revolutions. Observation uses
|
||
sin/cos so it's immune; reward wraps to `[-π, π]` internally.
|
||
- **StrictMode WebSocket double-mount** in dev — React's StrictMode mounts
|
||
the `useGameSocket` effect twice, so you'll see one benign
|
||
"closed before established" WebSocket warning per page load.
|
||
- **Single-player only** — one CLI subprocess per connection; multi-user
|
||
demo is fine at demo scale but there's no shared world.
|
||
|
||
## Debugging tips
|
||
|
||
- **Backend can't find CLI or ONNX at startup** — the repo-root walker looks
|
||
for `GameCli.sln`. Both `.sln` and `.slnx` formats work.
|
||
- **`dotnet test` in `Backend.Tests` uses the CLI** — publish first
|
||
(`dotnet publish GameCli -c Release -o publish/GameCli`) so
|
||
`EndToEndSmokeTests` can find the binary.
|
||
- **`torch.onnx.export` on torch 2.13+** — pass `dynamo=False` to force the
|
||
legacy TorchScript exporter unless `onnxscript` is installed.
|
||
- **Vite WS proxy failing** — verify with a Node script bypassing the proxy
|
||
(`ws://localhost:5100/ws/game` direct). If direct works but proxy doesn't,
|
||
check `Frontend/vite.config.ts` proxy block.
|