Design spec: surface reply and reply-with-quote context to the agent
This commit is contained in:
EugeneTes
192
docs/superpowers/specs/2026-09-22-reply-context-design.md
Normal file
192
docs/superpowers/specs/2026-09-22-reply-context-design.md
Normal file
@@ -0,0 +1,192 @@
|
|||||||
|
# Reply and "reply with quote" context for the agent
|
||||||
|
|
||||||
|
## Goal
|
||||||
|
|
||||||
|
Give the agent visibility into which message the owner is replying to. Telegram already sends this on every reply — `message.reply_to_message` for a plain reply and `message.quote` (Bot API 7.0+) when the user highlighted a specific substring — but `normalizeMessage` currently drops both. As a result, when the owner replies to one of the bot's older answers or their own earlier message, the agent has no idea what "this" refers to.
|
||||||
|
|
||||||
|
This is the mirror image of the existing `[Forwarded from …]` header the agent already gets for forwarded messages.
|
||||||
|
|
||||||
|
## Non-goals
|
||||||
|
|
||||||
|
- **Rewinding the conversation.** We do not seek the replied-to turn inside the agent's own history and re-anchor the conversation there. The agent still runs strictly forward; the reply just becomes context injected into the next turn.
|
||||||
|
- **`external_reply`.** Telegram's Bot API 7.0 also delivers `external_reply` (a reply to a message in a different chat, e.g. a channel post from a discussion group). The bot only runs in a private 1:1 chat with the owner, so `external_reply` shouldn't arise. If it ever does, we'll surface it later.
|
||||||
|
- **Re-attaching media from the replied-to message.** If the owner replies to a photo they sent yesterday, we describe it (`(photo)`) instead of re-downloading and re-attaching bytes. The agent already has that photo in its transcript when the reply is to its own turn; when the reply is to the owner's own turn, the bytes were already used at that time.
|
||||||
|
- **Quote-with-entities rendering.** `quote.entities` may carry bold/italic/URL markers on the highlighted range; we treat the quote as plain text and drop entity info. Cheap, and models understand plain quoted text fine.
|
||||||
|
- **Reply metadata in outbound.** Telegram lets a bot post a message *as a reply* to another (`reply_parameters`). Not adding that here — the outbound path is unchanged.
|
||||||
|
|
||||||
|
## Architecture
|
||||||
|
|
||||||
|
All changes live in `src/telegram/inbound.ts` (plus test file and a two-line change in `bridge.ts` to pass the bot user id through).
|
||||||
|
|
||||||
|
### Data model additions
|
||||||
|
|
||||||
|
Extend `InboundMessage`:
|
||||||
|
|
||||||
|
```ts
|
||||||
|
export interface InboundMessage {
|
||||||
|
// …existing fields…
|
||||||
|
reply_to_message?: RepliedMessage;
|
||||||
|
quote?: { text: string; is_manual?: boolean };
|
||||||
|
}
|
||||||
|
|
||||||
|
export interface RepliedMessage {
|
||||||
|
from?: { id: number; is_bot?: boolean };
|
||||||
|
text?: string;
|
||||||
|
caption?: string;
|
||||||
|
photo?: unknown[];
|
||||||
|
voice?: unknown;
|
||||||
|
audio?: { file_name?: string };
|
||||||
|
document?: { file_name?: string };
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
`RepliedMessage` is deliberately a loose subset of Telegram's `Message` — we only read the shape we care about, matching the codebase's existing pattern of declaring hand-narrowed message types instead of pulling grammy's `Message` type in wholesale.
|
||||||
|
|
||||||
|
Extend `InboundDeps`:
|
||||||
|
|
||||||
|
```ts
|
||||||
|
export interface InboundDeps {
|
||||||
|
// …existing fields…
|
||||||
|
/** The bot's own user id, for labelling replies to the bot vs the owner's own messages. */
|
||||||
|
botUserId?: number;
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
Optional so existing tests don't have to supply it; when omitted, replies fall back to "your own earlier message" wording only when `from.is_bot === false`, otherwise the neutral wording "the earlier message".
|
||||||
|
|
||||||
|
### Header format
|
||||||
|
|
||||||
|
Mirrors the existing `[Forwarded from …]` convention. One line, English, quoted content in double quotes, truncated with `…` if the content exceeds 500 characters.
|
||||||
|
|
||||||
|
| Situation | Header |
|
||||||
|
| --- | --- |
|
||||||
|
| Plain reply to the bot | `[Reply to your message: "…"]` |
|
||||||
|
| Plain reply to owner's own message | `[Reply to your own earlier message: "…"]` |
|
||||||
|
| Reply-with-quote to the bot | `[Reply quoting your message: "<quote>"]` |
|
||||||
|
| Reply-with-quote to owner's own | `[Reply quoting your own earlier message: "<quote>"]` |
|
||||||
|
| Reply to media with no text | `[Reply to your message: "(photo)"]` etc. |
|
||||||
|
| Reply to media WITH caption | Caption is used as the "…" content; media type is not mentioned separately. Keeps the header short and lets the model see what the owner actually wrote. |
|
||||||
|
|
||||||
|
Media descriptions (used only when there is no text and no caption on the replied-to message), checked in this fixed order — first match wins:
|
||||||
|
1. photo present → `(photo)`
|
||||||
|
2. voice present → `(voice message)`
|
||||||
|
3. audio present → `(audio: <file_name>)` or `(audio)` if unnamed
|
||||||
|
4. document present → `(document: <file_name>)` or `(document)` if unnamed
|
||||||
|
5. none of the above → `(message)`
|
||||||
|
|
||||||
|
Truncation: if the chosen source text is longer than 500 characters, keep the first 499 and append `…`. Applies to plain-reply text and to the quoted substring alike.
|
||||||
|
|
||||||
|
### Header content selection
|
||||||
|
|
||||||
|
For a plain reply, prefer in order: `text` → `caption` → media description.
|
||||||
|
|
||||||
|
For a reply with quote, always use `quote.text` verbatim (that's the whole point of quoting). If somehow `quote.text` is empty, fall through as if it were a plain reply.
|
||||||
|
|
||||||
|
### From-labelling
|
||||||
|
|
||||||
|
Given `botUserId` and `reply.from?.id`:
|
||||||
|
|
||||||
|
- `from.id === botUserId` (or `from.is_bot === true` when `botUserId` isn't supplied) → "your" wording
|
||||||
|
- `from` present and matches the sender of the incoming message (i.e. the owner) → "your own earlier" wording
|
||||||
|
- `from` absent → "the earlier" wording (a safe default that reads naturally)
|
||||||
|
|
||||||
|
The owner sender-id doesn't need to be threaded in explicitly: the bot runs in a private chat with a single owner id, so if `from.is_bot !== true` and it's not the bot, it is by definition the owner.
|
||||||
|
|
||||||
|
### Ordering with forwards
|
||||||
|
|
||||||
|
A message can be a reply AND a forward simultaneously (owner forwards a channel post and replies to one of the bot's earlier questions). The reply header comes first because it's the local context ("this is a follow-up on _that_"); the forward header comes second because it describes the payload's provenance:
|
||||||
|
|
||||||
|
```
|
||||||
|
[Reply to your message: "which article did you mean?"]
|
||||||
|
[Forwarded from Daily News]
|
||||||
|
Story about geothermal…
|
||||||
|
```
|
||||||
|
|
||||||
|
For a forwarded photo-only reply:
|
||||||
|
|
||||||
|
```
|
||||||
|
[Reply to your message: "which of these?"]
|
||||||
|
[Forwarded from @alice]
|
||||||
|
(photo attached)
|
||||||
|
```
|
||||||
|
|
||||||
|
### Interaction with debouncer
|
||||||
|
|
||||||
|
None. `normalizeMessage` still returns a single `AgentInput` per Telegram message; the debouncer merges them by concatenating text with `\n\n` and unioning images, unchanged.
|
||||||
|
|
||||||
|
### Wiring in `bridge.ts`
|
||||||
|
|
||||||
|
Grammy has already called `bot.init()` by the time `bot.start()` returns and it's called explicitly at startup in `main.ts`. `bot.botInfo.id` is safe to read once the app is running. Pass it into the deps:
|
||||||
|
|
||||||
|
```ts
|
||||||
|
const result = await normalizeMessage(ctx.message, {
|
||||||
|
download: (fileId) => downloadFile(bot, config.telegramBotToken, fileId, deps.fetchFn),
|
||||||
|
transcriber: deps.transcriber,
|
||||||
|
workspaceDir: config.workspaceDir,
|
||||||
|
botUserId: bot.botInfo?.id,
|
||||||
|
});
|
||||||
|
```
|
||||||
|
|
||||||
|
`botInfo` is populated by grammy after `bot.init()`. If it's undefined at first message (shouldn't happen after `start`), the header still renders correctly using the `is_bot` fallback described above.
|
||||||
|
|
||||||
|
## Data flow
|
||||||
|
|
||||||
|
```
|
||||||
|
owner: replies to a bot message "What's the CSV path?" with "workspace/foo.csv"
|
||||||
|
→ grammy delivers ctx.message with reply_to_message.text === "What's the CSV path?"
|
||||||
|
and reply_to_message.from.is_bot === true
|
||||||
|
→ normalizeMessage builds
|
||||||
|
"[Reply to your message: \"What's the CSV path?\"]\nworkspace/foo.csv"
|
||||||
|
→ debouncer flushes → host.send(AgentInput)
|
||||||
|
→ agent sees the header and knows this is an answer to its own earlier question
|
||||||
|
```
|
||||||
|
|
||||||
|
Reply-with-quote:
|
||||||
|
|
||||||
|
```
|
||||||
|
owner highlights "geothermal power plant" in a channel forward the bot sent yesterday
|
||||||
|
→ ctx.message.quote.text === "geothermal power plant"
|
||||||
|
→ header: "[Reply quoting your message: \"geothermal power plant\"]"
|
||||||
|
```
|
||||||
|
|
||||||
|
## Testing
|
||||||
|
|
||||||
|
Add cases to `src/telegram/inbound.test.ts`:
|
||||||
|
|
||||||
|
1. Plain reply to a bot message → header uses "your message"; body preserved.
|
||||||
|
2. Plain reply to owner's own earlier message (`reply.from.is_bot === false`) → header uses "your own earlier message".
|
||||||
|
3. Reply where `reply.from` is absent → header uses "the earlier message".
|
||||||
|
4. Reply with quote where `quote.text` is set → header uses "quoting" wording with the quote as content.
|
||||||
|
5. Reply with quote where the source is the owner's own message → "quoting your own earlier message".
|
||||||
|
6. Reply to a photo-only message (no caption) → content becomes `(photo)`.
|
||||||
|
7. Reply to a voice-only message → `(voice message)`.
|
||||||
|
8. Reply to a document with a filename → `(document: <name>)`.
|
||||||
|
9. Reply to a media message that also has a caption → caption is used (media descriptor is NOT injected).
|
||||||
|
10. Very long replied-to text (600 chars) → truncated to 499 + `…`.
|
||||||
|
11. Combined reply + forward → reply header first line, forward header second line, body third.
|
||||||
|
12. Missing `botUserId` in deps + `reply.from.is_bot === true` → still labelled "your message".
|
||||||
|
13. Reply carries no usable content at all (`{}` for reply_to_message) → header renders `(message)`.
|
||||||
|
|
||||||
|
Existing test cases keep passing (no `reply_to_message`/`quote` in fixtures → no header).
|
||||||
|
|
||||||
|
## Error handling
|
||||||
|
|
||||||
|
Header construction cannot fail — it's pure string work over fields we type-narrow ourselves. If Telegram delivers a shape we don't expect (e.g. a media type we don't check), we fall back to `(message)` rather than throwing. Nothing here can block delivery of the owner's actual message.
|
||||||
|
|
||||||
|
## README updates
|
||||||
|
|
||||||
|
Add one bullet under "Talking to it":
|
||||||
|
|
||||||
|
> Reply to any of my messages (or "reply with quote" a specific part) and I'll see what you're answering to — useful for coming back to something from earlier in the day.
|
||||||
|
|
||||||
|
Add a manual E2E item #19:
|
||||||
|
|
||||||
|
> Reply to one of the bot's earlier messages with "what did you mean by that?" → the answer references the specific message being replied to (not the most recent one).
|
||||||
|
|
||||||
|
## Rejected alternatives
|
||||||
|
|
||||||
|
- **Classic `> quoted line` prefix instead of a header.** Rejected: inconsistent with the codebase's existing `[Forwarded from …]` convention, and less unambiguous — the model would have to infer that the quote is context vs a copy-paste from the user.
|
||||||
|
- **Structured `<reply-context>` XML block.** Rejected: heavier than needed, and the existing forward header proves plain-English headers work well for Pi's default model.
|
||||||
|
- **Rewind the agent's session to the replied-to turn.** Rejected: complex (requires knowing where in the transcript that turn lives, and possibly forking history), and the plain header already gives the model enough to answer correctly.
|
||||||
|
- **Re-attach the replied-to image/document bytes.** Rejected: doubles bandwidth for something the agent already has in context (when replying to its own turn) or already consumed earlier (when replying to owner's turn). Describe, don't re-ship.
|
||||||
|
- **Thread `ownerUserId` through the deps.** Rejected: the bot runs in a single-owner private chat, so anything that isn't the bot is the owner. Bot id alone is enough for the labelling logic.
|
||||||
Reference in New Issue
Block a user