Files

EugeneTes 99dc906d4e Design spec: surface reply and reply-with-quote context to the agent

2026-09-22 08:11:40 +00:00

11 KiB

Raw Permalink Blame History

Reply and "reply with quote" context for the agent

Goal

Give the agent visibility into which message the owner is replying to. Telegram already sends this on every reply — message.reply_to_message for a plain reply and message.quote (Bot API 7.0+) when the user highlighted a specific substring — but normalizeMessage currently drops both. As a result, when the owner replies to one of the bot's older answers or their own earlier message, the agent has no idea what "this" refers to.

This is the mirror image of the existing [Forwarded from …] header the agent already gets for forwarded messages.

Non-goals

Rewinding the conversation. We do not seek the replied-to turn inside the agent's own history and re-anchor the conversation there. The agent still runs strictly forward; the reply just becomes context injected into the next turn.
external_reply. Telegram's Bot API 7.0 also delivers external_reply (a reply to a message in a different chat, e.g. a channel post from a discussion group). The bot only runs in a private 1:1 chat with the owner, so external_reply shouldn't arise. If it ever does, we'll surface it later.
Re-attaching media from the replied-to message. If the owner replies to a photo they sent yesterday, we describe it ((photo)) instead of re-downloading and re-attaching bytes. The agent already has that photo in its transcript when the reply is to its own turn; when the reply is to the owner's own turn, the bytes were already used at that time.
Quote-with-entities rendering. quote.entities may carry bold/italic/URL markers on the highlighted range; we treat the quote as plain text and drop entity info. Cheap, and models understand plain quoted text fine.
Reply metadata in outbound. Telegram lets a bot post a message as a reply to another (reply_parameters). Not adding that here — the outbound path is unchanged.

Architecture

All changes live in src/telegram/inbound.ts (plus test file and a two-line change in bridge.ts to pass the bot user id through).

Data model additions

Extend InboundMessage:

export interface InboundMessage {
  // …existing fields…
  reply_to_message?: RepliedMessage;
  quote?: { text: string; is_manual?: boolean };
}

export interface RepliedMessage {
  from?: { id: number; is_bot?: boolean };
  text?: string;
  caption?: string;
  photo?: unknown[];
  voice?: unknown;
  audio?: { file_name?: string };
  document?: { file_name?: string };
}

RepliedMessage is deliberately a loose subset of Telegram's Message — we only read the shape we care about, matching the codebase's existing pattern of declaring hand-narrowed message types instead of pulling grammy's Message type in wholesale.

Extend InboundDeps:

export interface InboundDeps {
  // …existing fields…
  /** The bot's own user id, for labelling replies to the bot vs the owner's own messages. */
  botUserId?: number;
}

Optional so existing tests don't have to supply it; when omitted, replies fall back to "your own earlier message" wording only when from.is_bot === false, otherwise the neutral wording "the earlier message".

Header format

Mirrors the existing [Forwarded from …] convention. One line, English, quoted content in double quotes, truncated with … if the content exceeds 500 characters.

Situation	Header
Plain reply to the bot	`[Reply to your message: "…"]`
Plain reply to owner's own message	`[Reply to your own earlier message: "…"]`
Reply-with-quote to the bot	`[Reply quoting your message: "<quote>"]`
Reply-with-quote to owner's own	`[Reply quoting your own earlier message: "<quote>"]`
Reply to media with no text	`[Reply to your message: "(photo)"]` etc.
Reply to media WITH caption	Caption is used as the "…" content; media type is not mentioned separately. Keeps the header short and lets the model see what the owner actually wrote.

Media descriptions (used only when there is no text and no caption on the replied-to message), checked in this fixed order — first match wins:

photo present → (photo)
voice present → (voice message)
audio present → (audio: <file_name>) or (audio) if unnamed
document present → (document: <file_name>) or (document) if unnamed
none of the above → (message)

Truncation: if the chosen source text is longer than 500 characters, keep the first 499 and append …. Applies to plain-reply text and to the quoted substring alike.

Header content selection

For a plain reply, prefer in order: text → caption → media description.

For a reply with quote, always use quote.text verbatim (that's the whole point of quoting). If somehow quote.text is empty, fall through as if it were a plain reply.

From-labelling

Given botUserId and reply.from?.id:

from.id === botUserId (or from.is_bot === true when botUserId isn't supplied) → "your" wording
from present and matches the sender of the incoming message (i.e. the owner) → "your own earlier" wording
from absent → "the earlier" wording (a safe default that reads naturally)

The owner sender-id doesn't need to be threaded in explicitly: the bot runs in a private chat with a single owner id, so if from.is_bot !== true and it's not the bot, it is by definition the owner.

Ordering with forwards

A message can be a reply AND a forward simultaneously (owner forwards a channel post and replies to one of the bot's earlier questions). The reply header comes first because it's the local context ("this is a follow-up on that"); the forward header comes second because it describes the payload's provenance:

[Reply to your message: "which article did you mean?"]
[Forwarded from Daily News]
Story about geothermal…

For a forwarded photo-only reply:

[Reply to your message: "which of these?"]
[Forwarded from @alice]
(photo attached)

Interaction with debouncer

None. normalizeMessage still returns a single AgentInput per Telegram message; the debouncer merges them by concatenating text with \n\n and unioning images, unchanged.

Wiring in `bridge.ts`

Grammy has already called bot.init() by the time bot.start() returns and it's called explicitly at startup in main.ts. bot.botInfo.id is safe to read once the app is running. Pass it into the deps:

const result = await normalizeMessage(ctx.message, {
  download: (fileId) => downloadFile(bot, config.telegramBotToken, fileId, deps.fetchFn),
  transcriber: deps.transcriber,
  workspaceDir: config.workspaceDir,
  botUserId: bot.botInfo?.id,
});

botInfo is populated by grammy after bot.init(). If it's undefined at first message (shouldn't happen after start), the header still renders correctly using the is_bot fallback described above.

Data flow

owner: replies to a bot message "What's the CSV path?" with "workspace/foo.csv"
  → grammy delivers ctx.message with reply_to_message.text === "What's the CSV path?"
    and reply_to_message.from.is_bot === true
  → normalizeMessage builds
      "[Reply to your message: \"What's the CSV path?\"]\nworkspace/foo.csv"
  → debouncer flushes → host.send(AgentInput)
  → agent sees the header and knows this is an answer to its own earlier question

Reply-with-quote:

owner highlights "geothermal power plant" in a channel forward the bot sent yesterday
  → ctx.message.quote.text === "geothermal power plant"
  → header: "[Reply quoting your message: \"geothermal power plant\"]"

Testing

Add cases to src/telegram/inbound.test.ts:

Plain reply to a bot message → header uses "your message"; body preserved.
Plain reply to owner's own earlier message (reply.from.is_bot === false) → header uses "your own earlier message".
Reply where reply.from is absent → header uses "the earlier message".
Reply with quote where quote.text is set → header uses "quoting" wording with the quote as content.
Reply with quote where the source is the owner's own message → "quoting your own earlier message".
Reply to a photo-only message (no caption) → content becomes (photo).
Reply to a voice-only message → (voice message).
Reply to a document with a filename → (document: <name>).
Reply to a media message that also has a caption → caption is used (media descriptor is NOT injected).
Very long replied-to text (600 chars) → truncated to 499 + ….
Combined reply + forward → reply header first line, forward header second line, body third.
Missing botUserId in deps + reply.from.is_bot === true → still labelled "your message".
Reply carries no usable content at all ({} for reply_to_message) → header renders (message).

Existing test cases keep passing (no reply_to_message/quote in fixtures → no header).

Error handling

Header construction cannot fail — it's pure string work over fields we type-narrow ourselves. If Telegram delivers a shape we don't expect (e.g. a media type we don't check), we fall back to (message) rather than throwing. Nothing here can block delivery of the owner's actual message.

README updates

Add one bullet under "Talking to it":

Reply to any of my messages (or "reply with quote" a specific part) and I'll see what you're answering to — useful for coming back to something from earlier in the day.

Add a manual E2E item #19:

Reply to one of the bot's earlier messages with "what did you mean by that?" → the answer references the specific message being replied to (not the most recent one).

Rejected alternatives

Classic > quoted line prefix instead of a header. Rejected: inconsistent with the codebase's existing [Forwarded from …] convention, and less unambiguous — the model would have to infer that the quote is context vs a copy-paste from the user.
Structured <reply-context> XML block. Rejected: heavier than needed, and the existing forward header proves plain-English headers work well for Pi's default model.
Rewind the agent's session to the replied-to turn. Rejected: complex (requires knowing where in the transcript that turn lives, and possibly forking history), and the plain header already gives the model enough to answer correctly.
Re-attach the replied-to image/document bytes. Rejected: doubles bandwidth for something the agent already has in context (when replying to its own turn) or already consumed earlier (when replying to owner's turn). Describe, don't re-ship.
Thread ownerUserId through the deps. Rejected: the bot runs in a single-owner private chat, so anything that isn't the bot is the owner. Bot id alone is enough for the labelling logic.

11 KiB Raw Permalink Blame History