forked from admin/VolcanicAgenticCore
fix(telegram): address review findings
- A speech or download timeout (TaskCanceledException nobody asked for) is a failure with the user notice, not a silent shutdown; OpenAiSpeechToText maps it to "the request timed out."; UpdateRouter only treats the caller's cancellation as shutdown. - send: WorkspacePaths resolves the real path component by component (realpath), closing the directory-link-inside-target bypass; `.` reports a directory. - Image documents are attached inline only when their magic bytes are JPEG/PNG/GIF/WebP, under the detected type. - Audio goes to STT as an ASCII audio.<ext> name. - MCP Browser section mentions send only when it is active. - RunInOrderAsync takes a cancellation token; queued work is skipped once disposal gives up; SendReplyAsync stays private behind ReplyInOrderAsync. - Speech HttpClient handler recycles pooled connections; BuiltInTools.Send(); IExtensionHost.Channel is abstract; WorkspacePaths in its own file. - Tests: timeouts, symlink chains (Linux), sniffing, channel delivery failures, queued-work cancellation, gated voice ordering. Spec and docs updated.
This commit is contained in:
@@ -78,7 +78,7 @@ Configured values win over the variables; an empty variable counts as unset. The
|
||||
- Each chat has its own browser profile under `.volcanic/browser-profile`, so site logins survive idle restarts,
|
||||
`/new` and bot restarts. The agent's `bash` can read that profile, like everything else in its workspace.
|
||||
- The system prompt gets a short "Browser" section while the browser tools are active. It mentions `send` only when
|
||||
a `send` tool exists.
|
||||
a `send` tool is active.
|
||||
|
||||
### Deployment requirements
|
||||
|
||||
|
||||
@@ -194,8 +194,9 @@ callTimeoutSeconds: 120, startupTimeoutSeconds: 120 (the first npx run downloads
|
||||
- Don't enter passwords or payment details unless the user explicitly asks for that specific site.
|
||||
```
|
||||
|
||||
The `send` line is included only when a tool named `send` is registered (checked at prompt time through
|
||||
`GetAllTools()`), so the preset still reads correctly on hosts without attachments. Without `send`, that bullet
|
||||
The `send` line is included only when a tool named `send` is active (checked at prompt time through
|
||||
`GetActiveTools()`; a registered but inactive `send` isn't callable), so the preset still reads correctly on hosts
|
||||
without attachments. Without `send`, that bullet
|
||||
reads: "Files the browser saves (screenshots with a `filename` such as `browser/<name>.png`, PDFs, downloads) are
|
||||
under browser/ in the workspace."
|
||||
|
||||
|
||||
@@ -118,10 +118,14 @@ ExecutionMode: Sequential (two sends in one turn arrive in call order)
|
||||
Execution:
|
||||
|
||||
1. `ctx.Channel` is null → throw `send is not available: this session has no chat` (defensive; see registration).
|
||||
2. Resolve with the existing `Paths.Resolve(ctx.Cwd, path)`. Reject unless the full path is inside `ctx.Cwd` (compare
|
||||
with a trailing separator, case-insensitively on Windows). If the file or any parent is a symlink, resolve it with
|
||||
`ResolveLinkTarget(returnFinalTarget: true)` and apply the same check to the target. Errors are
|
||||
`path escapes the workspace: <path>`, `file not found: <path>` and `path is a directory, not a file: <path>`.
|
||||
2. Resolve with the existing `Paths.Resolve(ctx.Cwd, path)`. Reject unless the full path is inside `ctx.Cwd` or is
|
||||
`ctx.Cwd` itself (compare with a trailing separator, case-insensitively on Windows). Then resolve the real path the
|
||||
way `realpath(3)` does (`WorkspacePaths.RealPath`): walk it from the filesystem root and replace each link by its
|
||||
target as it is met, so links inside a link's target are followed too and a `..` in a target applies to the real
|
||||
parent. That real path must be inside the real path of `ctx.Cwd`. More than 40 links (a loop) also fails.
|
||||
(`ResolveLinkTarget(returnFinalTarget: true)` isn't enough: on Unix it follows only the last component of each hop,
|
||||
so `a/l -> ../d/secret` with `d -> /etc` would pass.) Errors are `path escapes the workspace: <path>`,
|
||||
`file not found: <path>` and `path is a directory, not a file: <path>` (also for `.`, the workspace itself).
|
||||
3. Caption longer than `Limits.MaxCaptionLength` → `caption is <n> characters; the limit is <m>. Shorten it or put
|
||||
the text in your reply.` The check runs before anything is sent, so a failure never delivers half an attachment.
|
||||
4. Kind:
|
||||
@@ -151,13 +155,16 @@ it (`chat.Channel`).
|
||||
- `Limits`: photo 10 MB, document 50 MB, caption 1024 (Bot API limits), in `TelegramLimits` constants.
|
||||
- **Ordering through the renderer.** Session events are rendered asynchronously from `ChatRenderer`'s queue. A file
|
||||
sent directly could overtake an assistant message that ended just before the tool ran. `ChatRenderer` gets
|
||||
`Task<T> RunInOrderAsync<T>(Func<CancellationToken, Task<T>> work)`. It queues a host-internal work item behind the
|
||||
pending events and completes when the item has run. The renderer marks the status message as buried afterwards, as
|
||||
`Task<T> RunInOrderAsync<T>(Func<CancellationToken, Task<T>> work, CancellationToken cancellationToken = default)`.
|
||||
It queues a host-internal work item behind the pending events and completes when the item has run. The caller's
|
||||
token cancels the returned task at once, even while the item still waits behind a slow upload; an item cancelled
|
||||
before it starts is skipped, and a running one sees the token through the one it's given. The renderer marks the status message as buried afterwards, as
|
||||
it does for replies, so the `⚙️ Working...` message moves back to the bottom on the next tool update.
|
||||
`SendTextAsync` and `SendFileAsync` both go through it. The event channel becomes a channel of render items (an
|
||||
event or a work item). A work item still queued when the renderer is disposed is cancelled. `ChatRenderer.SayAsync(text)`
|
||||
(a `RunInOrderAsync` around `TelegramSender.SendTextAsync`) sends plain-text notices and the `🎤` echo. The reply path
|
||||
(`ChatRenderer.SendReplyAsync`) returns the first message id and whether every chunk was delivered.
|
||||
event or a work item). Disposing the renderer lets it drain its queue for up to 5 s; a work item still queued after
|
||||
that, or queued after disposal, is cancelled without running. `ChatRenderer.SayAsync(text, ct)` (a `RunInOrderAsync`
|
||||
around `TelegramSender.SendTextAsync`) sends plain-text notices and the `🎤` echo. `ChatRenderer.ReplyInOrderAsync`
|
||||
runs the (private) reply path on the loop and returns the first message id and whether every chunk was delivered.
|
||||
- **Files.** `TelegramSender.SendFileAsync(chatId, path, asPhoto, caption, ct) → Task<FileSendResult>`, with
|
||||
`FileSendResult(int? MessageId, string? Error, bool PhotoRejected)`. It follows the sender's never-throw convention.
|
||||
Each attempt opens a fresh `FileStream` and uses `InputFile.FromStream(stream, Path.GetFileName(path))` with
|
||||
@@ -218,7 +225,12 @@ Router changes (`UpdateRouter.OnMessageAsync`), on top of the message-context sp
|
||||
`[File received: inbox/2026-09-26-report.pdf (application/pdf, 1.2 MB)]`. The path is relative to the workspace
|
||||
and uses `/`. Sizes read `N bytes`, `N.N KB` or `N.N MB`.
|
||||
5. Image documents: when `MimeType` is `image/jpeg|png|webp|gif` and the size is ≤ `Telegram:InlineImageMaxBytes`
|
||||
(default 3 750 000), the saved bytes are also attached as `ImageContent(base64, mime)`. Larger images are
|
||||
(default 3 750 000), the saved bytes are also attached as `ImageContent(base64, mime)`. Documents aren't
|
||||
re-encoded by Telegram, so the declared type can be wrong (HEIC named `.png`, a truncated file); a provider rejects
|
||||
such an image, and the persisted prompt would then break every later turn. So the leading bytes decide: JPEG
|
||||
`FF D8 FF`, PNG `89 50 4E 47 0D 0A 1A 0A`, GIF `47 49 46 38` or WebP `RIFF....WEBP`, and the image is attached
|
||||
under the type its bytes show. Bytes that are none of these make the file path-only, with the suffix
|
||||
` (not shown inline: the contents are not a JPEG, PNG, WebP or GIF image)`. Larger images are
|
||||
path-only, and the line gets a parenthesised suffix after the size:
|
||||
`[File received: inbox/x.png (image/png, 5.0 MB) (too large to show inline; downscale it with bash if you need to see it)]`.
|
||||
|
||||
@@ -231,10 +243,15 @@ Router changes (`UpdateRouter.OnMessageAsync`), on top of the message-context sp
|
||||
refused it).
|
||||
3. Voice is downloaded into memory; a failed download sends `I could not download that file.` and the message isn't
|
||||
prompted. Audio is read back from the file step 2 saved, so it's downloaded only once. Voice arrives as `.oga`,
|
||||
which OpenAI rejects, so it's renamed to `voice.ogg` (`audio/ogg`). Audio keeps `FileName ?? "audio"` and its
|
||||
`MimeType`.
|
||||
which OpenAI rejects, so it's renamed to `voice.ogg` (`audio/ogg`). OpenAI picks the format from the file
|
||||
extension, so audio goes as an ASCII `audio.<ext>`: the original extension when OpenAI knows it (`flac`, `m4a`,
|
||||
`mp3`, `mp4`, `mpeg`, `mpga`, `ogg`, `wav`, `webm`; `.oga`/`.opus` become `.ogg`), else one from `MimeType`
|
||||
(`mp3` when unknown), with its `MimeType`. A missing or non-ASCII name would otherwise lose the extension.
|
||||
4. `Transcript = await stt.TranscribeAsync(bytes, fileName, mime, ct)`, trimmed. Empty → notice `I couldn't make
|
||||
out any words in that voice message.` An exception → notice `Could not transcribe the voice message: <reason>`.
|
||||
That includes a cancellation the message's own token didn't cause, such as the `HttpClient` timeout; only a
|
||||
shutdown (the caller's token) ends composing silently. The same rule holds for downloads, and `UpdateRouter`
|
||||
treats any other `OperationCanceledException` from `ComposeAsync` as a failure (`I could not read that message.`).
|
||||
5. Echo `🎤 <transcript>` as plain text, cut to 4096 characters with `…`.
|
||||
6. Prompt text: the caption (if any), then `[Voice message] <transcript>`. For `Audio`, the file is also saved to
|
||||
the inbox like a document and gets its own `[File received: …]` line: it's something the user might want
|
||||
@@ -261,7 +278,13 @@ public interface ISpeechToText
|
||||
as given), `model`, `response_format=json`, and `language` when configured.
|
||||
- Non-2xx → `SpeechToTextException("OpenAI transcription failed: HTTP <status> <first 200 chars of body>")`.
|
||||
A non-JSON body or a missing `text` gives its own message. The key never appears in messages or logs.
|
||||
- The client timeout (a `TaskCanceledException` while the caller's token isn't cancelled) →
|
||||
`SpeechToTextException("OpenAI transcription failed: the request timed out.")`; a transport failure
|
||||
(`HttpRequestException`) → the same prefix plus its message. The caller's own cancellation still propagates.
|
||||
- `HttpClient` comes from `AddHttpClient<ISpeechToText, OpenAiSpeechToText>()` (package `Microsoft.Extensions.Http`), with a 2-minute timeout.
|
||||
The singleton `InboundMedia` keeps that typed client for the process lifetime, so the factory never rotates its
|
||||
handler: the primary handler is a `SocketsHttpHandler` with `PooledConnectionLifetime = 5 min` (and an infinite
|
||||
handler lifetime), so DNS changes are still picked up.
|
||||
|
||||
`SpeechOptions`, the `Telegram:Speech` section:
|
||||
|
||||
@@ -274,7 +297,7 @@ public interface ISpeechToText
|
||||
|
||||
The environment fallbacks (`OPENAI_API_KEY`, `STT_MODEL`) are read in `TelegramOptions.ApplyEnvironment`, next to
|
||||
the existing `TELEGRAM_*` shortcuts, and only when the `Telegram:Speech` setting is empty. An empty variable counts
|
||||
as unset (the Docker compose file passes both through, empty by default). The OpenAI provider
|
||||
as unset (a deployment, such as a Docker compose file, may pass both through empty). The OpenAI provider
|
||||
extension, if loaded, reads `OPENAI_API_KEY` on its own; the two don't share state.
|
||||
`Program.cs` registers `ISpeechToText` only when a key is present, and logs `Speech to text: <model>` or
|
||||
`Speech to text: off (no OPENAI_API_KEY)`. `OPENAI_API_KEY` isn't on the Claude Code provider's denied-variable
|
||||
@@ -304,7 +327,8 @@ accepts the consequence that the agent doesn't see them.
|
||||
| file > 20 MB sent to the bot | size notice, nothing reaches the agent |
|
||||
| download fails | `I could not download that file.`, partial file deleted, no prompt |
|
||||
| voice with no STT configured | not-configured notice; caption (if any) is still prompted |
|
||||
| STT HTTP/parse error | `Could not transcribe the voice message: …`, no prompt unless there's a caption |
|
||||
| STT HTTP/parse error or timeout | `Could not transcribe the voice message: …`, no prompt unless there's a caption (an audio file's `[File received: …]` line is still prompted) |
|
||||
| image document whose bytes aren't JPEG/PNG/WebP/GIF | saved and prompted path-only, with the "not shown inline" note |
|
||||
| `send` path outside the workspace / missing / directory | error tool result, nothing sent |
|
||||
| `send` over the photo / document limit | error tool result naming the limit |
|
||||
| caption over 1024 | error tool result, nothing sent |
|
||||
@@ -341,7 +365,9 @@ xunit, no network, no paid API.
|
||||
- `auto` `.png` passes `Auto`; `auto` `.pdf` and `.gif` pass `Document`; explicit `photo`/`document` are honoured.
|
||||
- An `auto` image over the photo limit goes as `Document`; an explicit `photo` over the limit fails.
|
||||
- A document over the limit, a caption over the limit, `../x`, an absolute path outside, and a symlink inside the
|
||||
workspace pointing outside all fail, and the channel is never called.
|
||||
workspace pointing outside all fail, and the channel is never called. So do (Unix only) a link whose target runs
|
||||
through a directory link pointing outside (`a/l -> ../d/secret`, `d -> <outside>`), a link whose target has a `..`
|
||||
after a directory link (`l -> x/../secret`, `x -> <outside>/deep`) and a link loop. `.` is a directory.
|
||||
- Missing file and directory errors. Absolute paths inside the workspace are accepted.
|
||||
- A `ChatDeliveryException` becomes an error tool result with its message.
|
||||
- Result text and details. Sparse files (`FileStream.SetLength`) stand in for big files.
|
||||
@@ -374,9 +400,9 @@ xunit, no network, no paid API.
|
||||
note.
|
||||
- Voice with STT: the `🎤` echo comes before the answer, the prompt contains `[Voice message] …`, and the file name
|
||||
handed to STT ends in `.ogg`. A 5000-character transcript echo is cut to 4096.
|
||||
- Voice without STT: the not-configured notice, and no run. STT throwing: the failure notice, and no run.
|
||||
- Voice without STT: the not-configured notice, and no run. STT throwing, or timing out (a `TaskCanceledException` nobody asked for): the failure notice, and no run.
|
||||
- Ordering: a voice note followed immediately by text produces one prompt with the transcript before the text (the
|
||||
fake STT is delayed past the debounce window). `/stop` sent during a slow transcription is handled at once.
|
||||
fake STT is held on a gate while a `FakeTimeProvider` moves well past the debounce window, and nothing is prompted until it is released). `/stop` sent during a slow transcription is handled at once.
|
||||
- A document with a `/new` caption doesn't run `/new`. A voice note doesn't answer a pending `InputAsync`.
|
||||
- A sticker gets the "I can read…" reply.
|
||||
- `send` end to end: the faux provider calls `send` for `out/chart.png`, a `sendPhoto` upload with the caption
|
||||
@@ -389,7 +415,7 @@ xunit, no network, no paid API.
|
||||
**Speech (`OpenAiSpeechToTextTests`)** with a stub `HttpMessageHandler`: the multipart form has `file` (name and
|
||||
content type), `model` and `response_format=json`, plus `language` when set; the bearer header is present; the
|
||||
transcript is trimmed; non-2xx, non-JSON and missing `text` each give their own message, which never contains the
|
||||
key.
|
||||
key. A client timeout is a `SpeechToTextException`; the caller's cancellation stays a cancellation.
|
||||
|
||||
## Out of scope
|
||||
|
||||
|
||||
@@ -119,8 +119,8 @@ question in that chat (for example from an extension) replaces the pending promp
|
||||
- **Documents, videos and audio files** are saved to `inbox/<yyyy-MM-dd>-<name>` in the chat's workspace (a number is
|
||||
added when the name is taken: `-1`, `-2`, ...). The agent gets the caption plus a line such as
|
||||
`[File received: inbox/2026-09-26-report.pdf (application/pdf, 1.2 MB)]`. An image sent "as file" (JPEG, PNG, WebP,
|
||||
GIF) up to 3.75 MB (`Telegram:InlineImageMaxBytes`) is also shown to the model; a larger one reaches it as a path
|
||||
only.
|
||||
GIF) up to 3.75 MB (`Telegram:InlineImageMaxBytes`) is also shown to the model; a larger one, or one whose contents
|
||||
aren't really one of those formats, reaches it as a path only.
|
||||
- **Voice messages and audio files** are transcribed with OpenAI (`gpt-4o-transcribe` by default). The bot echoes
|
||||
`🎤 <transcript>` and the agent gets `[Voice message] <transcript>` (an audio file also gets its `[File received: …]`
|
||||
line). Without an API key the bot replies with how to set one up (`OPENAI_API_KEY`).
|
||||
|
||||
Reference in New Issue
Block a user