# Experiment_ReinforcementLearning — cursor-following lunar lander

A cursor-following lunar lander demo: user moves the mouse, a PPO-controlled
ship chases it against gravity. Deliberate strict separation between game
logic, backend, frontend, and training.

Live repo: `https://gitea.tes.gd/admin/Experiment_ReinforcementLearning`

## Architecture

Four processes with narrow, stable interfaces:

```
[React UI] ──WebSocket──▶ [ASP.NET backend] ──stdin/stdout──▶ [Game CLI (.NET)]
                                                                     ▲
                                                                     │ same protocol
                                                                     │
                                              [Python PPO trainer] ──┘
```

- **`GameCli/`** — C# .NET 8 console app. Stateless step-driven executable.
  Reads one action per line on stdin, writes one observation JSON per line on
  stdout. Owns physics, reward, observation. No networking, no UI knowledge.
- **`Backend/`** — ASP.NET Core Web API. WebSocket at `/ws/game`. Per
  connection: spawns one `GameCli` subprocess, runs a 50 Hz tick loop that
  calls the singleton `PolicyRunner` (ONNX) for each action.
- **`Frontend/`** — Vite + React + TypeScript. Opens a WebSocket via a
  same-origin proxy (`/ws` → `localhost:5100`), sends `mousemove` as world
  coords, renders the ship on a `<canvas>`.
- **`Training/`** — Python. `LanderCliEnv` wraps the **same** `GameCli` binary
  as a `gym.Env`, so training physics and inference physics are identical by
  construction. Produces `models/ppo_lander.onnx`.

Design spec: `docs/superpowers/specs/2026-07-17-cursor-following-lander-design.md`
Implementation plans: `docs/superpowers/plans/2026-07-17-{01-04}-*.md`

## Build & test

Everything runs from the repo root.

```bash
# Game CLI (25 xUnit tests)
dotnet test --filter FullyQualifiedName~GameCli.Tests

# Backend (8 xUnit tests — includes end-to-end WebSocket smoke)
dotnet publish GameCli/GameCli.csproj -c Release -o publish/GameCli
dotnet test --filter FullyQualifiedName~Backend.Tests

# Training (11 pytest tests)
source .venv/bin/activate
cd Training && pytest && cd ..

# Frontend
cd Frontend && npx tsc --noEmit && npm run build && cd ..
```

## Run the demo locally

```bash
# 1. Publish CLI (idempotent)
dotnet publish GameCli/GameCli.csproj -c Release -o publish/GameCli

# 2. Start backend (loads models/ppo_lander.onnx at startup)
ASPNETCORE_URLS=http://localhost:5100 \
  dotnet run --project Backend --no-launch-profile

# 3. In another shell, start Vite dev server
cd Frontend && npm run dev  # → http://localhost:5173/
```

Open `http://localhost:5173/`. Status bar top-left shows connection state and
step counter; blue crosshair follows your mouse; white triangle is the ship.

## Wire protocol reference

**CLI stdio** (line-delimited JSON):

- CLI startup: `{"init":{"world":[1,1],"dt":0.02,"obs_dim":7,"n_actions":4}}`
- Step in: `{"action":<0-3>,"target":[x,y]}`
- Reset in: `{"cmd":"reset","seed":<int>?}`
- Step out: `{"obs":[7 floats],"state":{x,y,angle,engine},"reward":<f>,"done":<b>,"step":<i>}`
- Actions: `0`=noop, `1`=left thruster, `2`=main engine, `3`=right thruster.
- Observation: `[dx, dy, vx, vy, sin(angle), cos(angle), angular_velocity]`.

**WebSocket** (`ws://localhost:5100/ws/game` or same-origin via Vite proxy):

- Client → server: `{"type":"cursor","x":<0-1>,"y":<0-1>}`
- Server → client (on connect): `{"type":"init","world":[1,1]}`
- Server → client (each 20 ms tick):
  `{"type":"state","x":..,"y":..,"angle":..,"engine":..,"target":[..,..],"step":..}`

Both are mirrored in code (`GameCli/Protocol.cs`, `Backend/GameProcess.cs`,
`Backend/WireMessages.cs`, `Frontend/src/protocol.ts`).

## Coordinate & physics conventions

- World is `[0, 1] × [0, 1]`. Y increases downward (screen-native).
- `angle = 0` = ship pointing up. Positive angle rotates clockwise.
- No walls, no ground. Ship can drift off-screen; only the reward pulls it back.
- Fixed `dt = 0.02` (50 Hz). Explicit Euler integration.
- Physics constants live in `GameCli/Physics.cs`; reward constants in
  `GameCli/Reward.cs`. All `public const`, easy to tune.

## PPO training

```bash
source .venv/bin/activate

# Smoke run (~30 s):
python Training/train.py --steps 10000 --n-envs 2 --use-dummy --max-episode-steps 200

# Real training (~hours on CPU):
python Training/train.py --steps 2000000 --n-envs 8

# Export to ONNX (backend reads this file):
python Training/export_onnx.py \
  --checkpoint checkpoints/ppo_lander_final.zip \
  --out models/ppo_lander.onnx
```

Backend loads `models/ppo_lander.onnx` at startup and fails fast if it's
missing. Restart the backend after re-exporting to pick up the new weights.

### Current state of the model

`models/ppo_lander.onnx` was trained for only 10k PPO steps as a
pipeline smoke — essentially random with slight bias. The ship falls off the
world in a few seconds under this policy. To see actual cursor-following
behavior, run the 2M-step training pass above (see PPO training section).

## Config paths & env vars

Backend resolves paths at startup by walking up from `AppContext.BaseDirectory`
to find `GameCli.sln` (the repo-root marker), then joining a relative config
path.

- `Lander:CliPath` (default `publish/GameCli/GameCli`)
- `Lander:ModelPath` (default `models/ppo_lander.onnx`)
- Env-var overrides: `LANDER__CliPath`, `LANDER__ModelPath` (ASP.NET
  double-underscore convention).

## Repo conventions

- Every commit lands on `main` (no PR flow — greenfield project).
- Author: `meelstorm <meelstorm@gmail.com>`. Use
  `-c user.email=meelstorm@gmail.com -c user.name=meelstorm` when committing.
- Conventional commit prefixes: `feat(gamecli):`, `feat(backend):`,
  `feat(training):`, `feat(frontend):`, `fix(...)`, `test(...)`, `chore:`.
- `.gitignore` covers `bin/`, `obj/`, `node_modules/`, `dist/`, `publish/`,
  `.venv/`, `checkpoints/`, `models/`, `tensorboard/`, `*.onnx`.
- The trained model is NOT in git — it's a build artifact.

## Known limitations & future work

- **Model is undertrained** — see "Current state of the model" above.
- **No walls** — a badly-trained policy loses to gravity within a second and
  the ship drifts off-screen forever. Design choice: reward should be enough
  to keep a well-trained policy in-world.
- **Angle wrap in reward is applied** but not in `Physics.Step` — the ship's
  `angle` field grows unbounded across many revolutions. Observation uses
  sin/cos so it's immune; reward wraps to `[-π, π]` internally.
- **StrictMode WebSocket double-mount** in dev — React's StrictMode mounts
  the `useGameSocket` effect twice, so you'll see one benign
  "closed before established" WebSocket warning per page load.
- **Single-player only** — one CLI subprocess per connection; multi-user
  demo is fine at demo scale but there's no shared world.

## Debugging tips

- **Backend can't find CLI or ONNX at startup** — the repo-root walker looks
  for `GameCli.sln`. Both `.sln` and `.slnx` formats work.
- **`dotnet test` in `Backend.Tests` uses the CLI** — publish first
  (`dotnet publish GameCli -c Release -o publish/GameCli`) so
  `EndToEndSmokeTests` can find the binary.
- **`torch.onnx.export` on torch 2.13+** — pass `dynamo=False` to force the
  legacy TorchScript exporter unless `onnxscript` is installed.
- **Vite WS proxy failing** — verify with a Node script bypassing the proxy
  (`ws://localhost:5100/ws/game` direct). If direct works but proxy doesn't,
  check `Frontend/vite.config.ts` proxy block.