diff --git a/docs/superpowers/specs/2026-09-22-reply-context-design.md b/docs/superpowers/specs/2026-09-22-reply-context-design.md new file mode 100644 index 0000000..e877634 --- /dev/null +++ b/docs/superpowers/specs/2026-09-22-reply-context-design.md @@ -0,0 +1,192 @@ +# Reply and "reply with quote" context for the agent + +## Goal + +Give the agent visibility into which message the owner is replying to. Telegram already sends this on every reply — `message.reply_to_message` for a plain reply and `message.quote` (Bot API 7.0+) when the user highlighted a specific substring — but `normalizeMessage` currently drops both. As a result, when the owner replies to one of the bot's older answers or their own earlier message, the agent has no idea what "this" refers to. + +This is the mirror image of the existing `[Forwarded from …]` header the agent already gets for forwarded messages. + +## Non-goals + +- **Rewinding the conversation.** We do not seek the replied-to turn inside the agent's own history and re-anchor the conversation there. The agent still runs strictly forward; the reply just becomes context injected into the next turn. +- **`external_reply`.** Telegram's Bot API 7.0 also delivers `external_reply` (a reply to a message in a different chat, e.g. a channel post from a discussion group). The bot only runs in a private 1:1 chat with the owner, so `external_reply` shouldn't arise. If it ever does, we'll surface it later. +- **Re-attaching media from the replied-to message.** If the owner replies to a photo they sent yesterday, we describe it (`(photo)`) instead of re-downloading and re-attaching bytes. The agent already has that photo in its transcript when the reply is to its own turn; when the reply is to the owner's own turn, the bytes were already used at that time. +- **Quote-with-entities rendering.** `quote.entities` may carry bold/italic/URL markers on the highlighted range; we treat the quote as plain text and drop entity info. Cheap, and models understand plain quoted text fine. +- **Reply metadata in outbound.** Telegram lets a bot post a message *as a reply* to another (`reply_parameters`). Not adding that here — the outbound path is unchanged. + +## Architecture + +All changes live in `src/telegram/inbound.ts` (plus test file and a two-line change in `bridge.ts` to pass the bot user id through). + +### Data model additions + +Extend `InboundMessage`: + +```ts +export interface InboundMessage { + // …existing fields… + reply_to_message?: RepliedMessage; + quote?: { text: string; is_manual?: boolean }; +} + +export interface RepliedMessage { + from?: { id: number; is_bot?: boolean }; + text?: string; + caption?: string; + photo?: unknown[]; + voice?: unknown; + audio?: { file_name?: string }; + document?: { file_name?: string }; +} +``` + +`RepliedMessage` is deliberately a loose subset of Telegram's `Message` — we only read the shape we care about, matching the codebase's existing pattern of declaring hand-narrowed message types instead of pulling grammy's `Message` type in wholesale. + +Extend `InboundDeps`: + +```ts +export interface InboundDeps { + // …existing fields… + /** The bot's own user id, for labelling replies to the bot vs the owner's own messages. */ + botUserId?: number; +} +``` + +Optional so existing tests don't have to supply it; when omitted, replies fall back to "your own earlier message" wording only when `from.is_bot === false`, otherwise the neutral wording "the earlier message". + +### Header format + +Mirrors the existing `[Forwarded from …]` convention. One line, English, quoted content in double quotes, truncated with `…` if the content exceeds 500 characters. + +| Situation | Header | +| --- | --- | +| Plain reply to the bot | `[Reply to your message: "…"]` | +| Plain reply to owner's own message | `[Reply to your own earlier message: "…"]` | +| Reply-with-quote to the bot | `[Reply quoting your message: ""]` | +| Reply-with-quote to owner's own | `[Reply quoting your own earlier message: ""]` | +| Reply to media with no text | `[Reply to your message: "(photo)"]` etc. | +| Reply to media WITH caption | Caption is used as the "…" content; media type is not mentioned separately. Keeps the header short and lets the model see what the owner actually wrote. | + +Media descriptions (used only when there is no text and no caption on the replied-to message), checked in this fixed order — first match wins: +1. photo present → `(photo)` +2. voice present → `(voice message)` +3. audio present → `(audio: )` or `(audio)` if unnamed +4. document present → `(document: )` or `(document)` if unnamed +5. none of the above → `(message)` + +Truncation: if the chosen source text is longer than 500 characters, keep the first 499 and append `…`. Applies to plain-reply text and to the quoted substring alike. + +### Header content selection + +For a plain reply, prefer in order: `text` → `caption` → media description. + +For a reply with quote, always use `quote.text` verbatim (that's the whole point of quoting). If somehow `quote.text` is empty, fall through as if it were a plain reply. + +### From-labelling + +Given `botUserId` and `reply.from?.id`: + +- `from.id === botUserId` (or `from.is_bot === true` when `botUserId` isn't supplied) → "your" wording +- `from` present and matches the sender of the incoming message (i.e. the owner) → "your own earlier" wording +- `from` absent → "the earlier" wording (a safe default that reads naturally) + +The owner sender-id doesn't need to be threaded in explicitly: the bot runs in a private chat with a single owner id, so if `from.is_bot !== true` and it's not the bot, it is by definition the owner. + +### Ordering with forwards + +A message can be a reply AND a forward simultaneously (owner forwards a channel post and replies to one of the bot's earlier questions). The reply header comes first because it's the local context ("this is a follow-up on _that_"); the forward header comes second because it describes the payload's provenance: + +``` +[Reply to your message: "which article did you mean?"] +[Forwarded from Daily News] +Story about geothermal… +``` + +For a forwarded photo-only reply: + +``` +[Reply to your message: "which of these?"] +[Forwarded from @alice] +(photo attached) +``` + +### Interaction with debouncer + +None. `normalizeMessage` still returns a single `AgentInput` per Telegram message; the debouncer merges them by concatenating text with `\n\n` and unioning images, unchanged. + +### Wiring in `bridge.ts` + +Grammy has already called `bot.init()` by the time `bot.start()` returns and it's called explicitly at startup in `main.ts`. `bot.botInfo.id` is safe to read once the app is running. Pass it into the deps: + +```ts +const result = await normalizeMessage(ctx.message, { + download: (fileId) => downloadFile(bot, config.telegramBotToken, fileId, deps.fetchFn), + transcriber: deps.transcriber, + workspaceDir: config.workspaceDir, + botUserId: bot.botInfo?.id, +}); +``` + +`botInfo` is populated by grammy after `bot.init()`. If it's undefined at first message (shouldn't happen after `start`), the header still renders correctly using the `is_bot` fallback described above. + +## Data flow + +``` +owner: replies to a bot message "What's the CSV path?" with "workspace/foo.csv" + → grammy delivers ctx.message with reply_to_message.text === "What's the CSV path?" + and reply_to_message.from.is_bot === true + → normalizeMessage builds + "[Reply to your message: \"What's the CSV path?\"]\nworkspace/foo.csv" + → debouncer flushes → host.send(AgentInput) + → agent sees the header and knows this is an answer to its own earlier question +``` + +Reply-with-quote: + +``` +owner highlights "geothermal power plant" in a channel forward the bot sent yesterday + → ctx.message.quote.text === "geothermal power plant" + → header: "[Reply quoting your message: \"geothermal power plant\"]" +``` + +## Testing + +Add cases to `src/telegram/inbound.test.ts`: + +1. Plain reply to a bot message → header uses "your message"; body preserved. +2. Plain reply to owner's own earlier message (`reply.from.is_bot === false`) → header uses "your own earlier message". +3. Reply where `reply.from` is absent → header uses "the earlier message". +4. Reply with quote where `quote.text` is set → header uses "quoting" wording with the quote as content. +5. Reply with quote where the source is the owner's own message → "quoting your own earlier message". +6. Reply to a photo-only message (no caption) → content becomes `(photo)`. +7. Reply to a voice-only message → `(voice message)`. +8. Reply to a document with a filename → `(document: )`. +9. Reply to a media message that also has a caption → caption is used (media descriptor is NOT injected). +10. Very long replied-to text (600 chars) → truncated to 499 + `…`. +11. Combined reply + forward → reply header first line, forward header second line, body third. +12. Missing `botUserId` in deps + `reply.from.is_bot === true` → still labelled "your message". +13. Reply carries no usable content at all (`{}` for reply_to_message) → header renders `(message)`. + +Existing test cases keep passing (no `reply_to_message`/`quote` in fixtures → no header). + +## Error handling + +Header construction cannot fail — it's pure string work over fields we type-narrow ourselves. If Telegram delivers a shape we don't expect (e.g. a media type we don't check), we fall back to `(message)` rather than throwing. Nothing here can block delivery of the owner's actual message. + +## README updates + +Add one bullet under "Talking to it": + +> Reply to any of my messages (or "reply with quote" a specific part) and I'll see what you're answering to — useful for coming back to something from earlier in the day. + +Add a manual E2E item #19: + +> Reply to one of the bot's earlier messages with "what did you mean by that?" → the answer references the specific message being replied to (not the most recent one). + +## Rejected alternatives + +- **Classic `> quoted line` prefix instead of a header.** Rejected: inconsistent with the codebase's existing `[Forwarded from …]` convention, and less unambiguous — the model would have to infer that the quote is context vs a copy-paste from the user. +- **Structured `` XML block.** Rejected: heavier than needed, and the existing forward header proves plain-English headers work well for Pi's default model. +- **Rewind the agent's session to the replied-to turn.** Rejected: complex (requires knowing where in the transcript that turn lives, and possibly forking history), and the plain header already gives the model enough to answer correctly. +- **Re-attach the replied-to image/document bytes.** Rejected: doubles bandwidth for something the agent already has in context (when replying to its own turn) or already consumed earlier (when replying to owner's turn). Describe, don't re-ship. +- **Thread `ownerUserId` through the deps.** Rejected: the bot runs in a single-owner private chat, so anything that isn't the bot is the owner. Bot id alone is enough for the labelling logic.