Experiment_ReinforcementLearning/docs/superpowers/specs/2026-07-17-cursor-following-lander-design.md

Cursor-Following Lunar Lander — Design

Date: 2026-07-17 Status: Approved (ready for implementation planning)

Goal

Build a cursor-following lunar lander demo trained with PPO. A user opens a web page, moves the mouse, and a rocket ship — controlled by a trained neural network policy — chases the cursor by firing its main engine and side thrusters, fighting gravity as it goes.

The organizing constraint: strict separation between game logic and UI. The game engine is a standalone CLI process. It knows nothing about UIs, HTTP, or PPO. The exact same CLI is used both as the training environment for the Python PPO trainer and as the physics core the ASP.NET backend drives at runtime. Training physics and inference physics are identical by construction.

Non-goals

• No terrain, no ground, no landing pad, no touchdown detection. The task is continuous cursor-tracking; the ship never lands.
• No user "control modes" (no keyboard control, no scripted controller, no idle mode). At runtime the ship is always driven by the trained PPO policy trying to reach the cursor. In training, the Python trainer drives the CLI directly.
• No multi-player, no accounts, no persistence.

Architecture

Four processes, communicating over narrow interfaces:

[React UI] ──WebSocket──▶ [ASP.NET backend] ──stdin/stdout──▶ [Game CLI (.NET)]
                                                                     ▲
                                                                     │ same protocol
                                                                     │
                                              [Python PPO trainer] ──┘

• Game CLI (C#/.NET): stateless step-driven executable. Reads one action per line on stdin, writes one observation JSON per line on stdout. Owns physics.
• ASP.NET backend: owns one long-lived WebSocket per player. On connect, spawns one Game CLI subprocess and loads the trained PPO ONNX model. Each tick: reads latest cursor, runs the policy, steps the CLI, forwards state to the browser.
• React frontend: captures the mouse position, sends it to the backend, receives ship state, renders it on a canvas.
• Python PPO trainer: wraps the same Game CLI as a gym.Env, trains a PPO policy with stable-baselines3, exports the trained network to ONNX for the backend to load.

Game CLI

Protocol

Line-delimited JSON over stdio. One line in → one physics step → one line out.

Startup handshake (CLI emits one line on start):

{"init": {"world": [1.0, 1.0], "dt": 0.02, "obs_dim": 7, "n_actions": 4}}

Step input (client writes one line):

{"action": 2, "target": [0.35, 0.60]}

• action ∈ {0: noop, 1: left thruster, 2: main engine, 3: right thruster}
• target — current goal position in world coords, updated every step.

Step output (CLI writes one line):

{
  "obs":   [dx, dy, vx, vy, sin_theta, cos_theta, omega],
  "state": {"x": 0.51, "y": 0.63, "angle": -0.12, "engine": 2},
  "reward": -0.42,
  "done":   false,
  "step":   137
}

• obs — 7-dim vector fed to PPO. dx = target.x − ship.x, dy = target.y − ship.y. Angle is encoded as (sin, cos) to avoid wrap discontinuity. omega is angular velocity.
• state — raw pose the UI needs to render. Redundant with obs, but rendering shouldn't have to un-normalize.
• reward, done, step — for the trainer; UI ignores them.

Reset command (client writes one line):

{"cmd": "reset", "seed": 12345}

Reinitializes ship to a random start pose (using seed if provided), clears step counter, emits a fresh observation line.

Physics

• World is [0, 1] × [0, 1]. No walls; the ship can drift off-screen and is pulled back by the distance-to-cursor reward term.
• Constant downward gravity g (acceleration).
• Main engine (action 2): thrust force along +body-up, rotated by the ship's angle.
• Left / right thruster (actions 1 and 3): apply torque (to rotate the ship) and a small lateral impulse.
• Explicit Euler integration at fixed dt = 0.02 (50 Hz).
• Constants (gravity, thrust magnitude, torque, drag, engine noise) live in a single Physics.cs with named fields for easy tuning.

Reward

Per step:

r = -‖(dx, dy)‖                    // distance-to-cursor  (dominant)
    − lambda_v     · ‖(vx, vy)‖    // velocity penalty
    − lambda_theta · |angle|       // upright penalty
    − lambda_fuel  · engine_on     // fuel penalty

Initial coefficients: lambda_v = 0.1, lambda_theta = 0.1, lambda_fuel = 0.03. All in Physics.cs, tunable.

Termination

• done is never set on distance or angle; the hover task is continuous.
• The trainer imposes a max-steps cap (1000) via truncation.
• The CLI honors an explicit reset command from either the trainer or the backend.

Backend (ASP.NET)

Components

• GameProcess — thin wrapper over System.Diagnostics.Process launching the Game CLI. Exposes Task<Observation> StepAsync(int action, (float x, float y) target) and Task ResetAsync(). Owns stdin/stdout with line-based reader. One instance per player. Disposal kills the process.
• GameSession — one per connected browser. Holds a GameProcess, current cursor position (via a System.Threading.Channels mailbox), and the last observation. Runs a 50 Hz tick loop:
  1. Read latest cursor from mailbox.
  2. Ask PolicyRunner for an action given the last obs.
  3. await gameProcess.StepAsync(action, cursor).
  4. Push the returned state to the WebSocket send queue.
• GameHub — the WebSocket endpoint at /ws/game. On connect: create GameSession, start its tick loop. On message: parse cursor updates and route to the session. On disconnect: dispose session (kills CLI).
• PolicyRunner — single instance per backend process (inference is stateless; weights are shared across sessions). Loads models/ppo_lander.onnx on startup via Microsoft.ML.OnnxRuntime. Exposes int SelectAction(float[7] obs) that runs one inference and argmaxes the logits. If the model file is missing at startup, the backend fails fast — there is no fallback.

WebSocket wire format

• Client → server:
  • {"type": "cursor", "x": 0.42, "y": 0.68} — sent on mousemove, throttled to ~50 Hz on the client.
• Server → client:
  • {"type": "init", "world": [1.0, 1.0]} — sent once on connect (forwarded from the CLI handshake).
  • {"type": "state", "x": ..., "y": ..., "angle": ..., "engine": 2, "target": [...], "step": 137} — one per tick.

Concurrency

One CLI process per session, one ticker task per session. Sessions are fully independent. N players = N CLI processes = N ticker tasks. Adequate for demo scale.

Frontend (React)

Components

• useGameSocket hook — opens the WebSocket, parses incoming messages, exposes {init, state} as React state. Reconnects on close. Sends outgoing cursor messages.
• <LanderCanvas> — a <canvas> rendered each frame via requestAnimationFrame, reading the latest state from the socket hook. Draws:
  • Ship body (simple polygon), rotated by state.angle.
  • A flame under the active engine if state.engine != 0 (main = downward flare, left thruster = right-side flare, right thruster = left-side flare).
  • A crosshair at state.target (server-echoed cursor, so the user can see the target the server is actually using).
• useCursorSender hook — listens for mousemove on the canvas, converts pixel coords to world coords ([0, 1] × [0, 1]), sends via the socket. Throttled to ~50 Hz using a requestAnimationFrame timestamp gate.
• <App> — full-viewport canvas, wires the hooks, shows a small connection-status indicator.

Coordinate mapping

World is [0, 1] × [0, 1]. Canvas dimensions are the viewport. World is letterboxed into the viewport (black bars on the wider axis) so the ship never distorts.

Rendering

Plain Canvas 2D, no WebGL, no game engine. The scene is one ship + one dot + one flame. Draw the latest received state directly; no interpolation. Add interpolation later only if visible judder appears.

Build

Vite + React + TypeScript. In production, ASP.NET serves the built assets as static files.

Training pipeline (Python)

LanderCliEnv(gym.Env)

Wraps one C# CLI subprocess as a Gymnasium environment.

• __init__: Popen the CLI, read init handshake, set observation_space = Box(-inf, +inf, (7,)), action_space = Discrete(4).
• reset(seed): pick a random target uniformly in [0, 1] × [0, 1], send {"cmd": "reset", "seed": seed}, read observation, return it.
• step(action): send {"action": a, "target": [tx, ty]} (target fixed for the whole episode), read one line, return (obs, reward, terminated=False, truncated=(step>=1000), info).
• close: close stdin, wait for exit.

Training

• Vectorized rollout: SB3.SubprocVecEnv with N LanderCliEnv instances (N = CPU cores) → N CLI subprocesses.
• Algorithm: stable_baselines3.PPO with MlpPolicy, hidden layers [64, 64] as a starting point, otherwise stock SB3 hyperparameters.
• Budget: 2M steps as a first pass. Checkpoint to checkpoints/, log to TensorBoard, eval every N steps against a fixed set of eval targets (reproducible seed) for a stable success metric.
• Target sampling for training: each episode picks a random target once and keeps it fixed. This is simpler than curriculum-based moving targets and generalizes fine at inference, because the policy runs at higher frequency than a cursor moves.

ONNX export (export_onnx.py)

• Load the final SB3 checkpoint.
• Extract policy.mlp_extractor + policy.action_net into a single torch.nn.Module that takes 7-dim input and returns 4-dim action logits.
• torch.onnx.export(...) with a fixed opset, dynamic axes disabled (batch=1).
• Write to models/ppo_lander.onnx.
• Verify: load with onnxruntime, run on 100 random obs, assert outputs match SB3 within 1e-5.

Repo layout

GameCli/                # C# .NET console app — the game engine (CLI)
Backend/                # ASP.NET Core Web API — WebSocket hub + CLI orchestration + ONNX inference
Frontend/               # React + Vite + TypeScript — mouse capture + canvas rendering
Training/               # Python — LanderCliEnv, train.py, export_onnx.py, requirements.txt
models/                 # exported ONNX (gitignored; produced by training)
checkpoints/            # SB3 training checkpoints (gitignored)
docs/superpowers/specs/ # design docs

Testing strategy (high level)

• GameCli: unit tests for physics integration (deterministic given seed) and reward computation.
• Backend: integration test that spawns the real CLI subprocess and asserts a full tick round-trip.
• Training: smoke test that LanderCliEnv passes check_env from Gymnasium.
• Frontend: manual smoke — visual check that the ship renders and responds.

Detailed test cases live in the implementation plan.