forked from admin/VolcanicAgenticCore
docs(telegram): attachments, voice, send and IChatChannel
telegram-bot.md: what you can send, files from the agent, speech settings, security notes. ui-integration.md: document/voice input rows and a new section 5 on IChatChannel. architecture, extensions and CLAUDE.md list IChatChannel. Drop the interim paragraph from the message-context spec; mark the attachments spec implemented with resolved questions.
This commit is contained in:
EugeneTes
@@ -12,7 +12,8 @@ tests/* xunit
|
||||
|
||||
`Abstractions` is deliberately small and stable: messages and content blocks, `Model` and `IProvider`,
|
||||
the streaming event protocol, `ITool`, the extension API and its events, session entries and stores,
|
||||
`IUserInterface`, the `AgentSessionEvent` family a UI subscribes to, and the process contract in `Processes/`
|
||||
`IUserInterface`, `IChatChannel` (delivering files and texts into the conversation's chat), the `AgentSessionEvent`
|
||||
family a UI subscribes to, and the process contract in `Processes/`
|
||||
(`IProcessSpawnPolicy`, `ProcessSpawnRequest`/`ProcessSpawnContext`, `ProcessSpawnHook`, `ProcessSpawnEvent`,
|
||||
`ISecretRedactor`). Extensions reference only this assembly, which is what allows them to be loaded from separate
|
||||
directories.
|
||||
@@ -29,7 +30,7 @@ owns everything for that feature and exposes one entry type:
|
||||
|---|---|---|
|
||||
| `Models` | `ModelRegistry` | Provider registrations, `provider/id` lookup, API key resolution, transcript normalisation before a call |
|
||||
| `Sessions` | `SessionManager` | The entry tree of one session, branch/context building, `ISessionStore` implementations |
|
||||
| `Tools` | `ToolRegistry` | Known and active tools, argument validation, output truncation, the built-in tools |
|
||||
| `Tools` | `ToolRegistry` | Known and active tools, argument validation, output truncation, the built-in tools (`send` only for sessions with a channel) |
|
||||
| `Extensions` | `ExtensionRunner` | Activating extensions (two-phase), dispatching events, contexts, the event bus, assembly loading |
|
||||
| `Commands` | `CommandRegistry` | Slash commands |
|
||||
| `SystemPrompt` | `SystemPromptBuilder` | Base prompt, tools section, guidelines, project context files, environment |
|
||||
|
||||
@@ -115,7 +115,8 @@ fails the spawn); `Cancel`, `Block` and `MarkHandled` stop further handlers.
|
||||
status), `SessionManager` (read-only entries and branch), `ModelRegistry` (models and provider-neutral
|
||||
`CompleteAsync` for nested model calls), `Model`, `ThinkingLevel`, `Signal` (cancelled on abort), `IsIdle()`,
|
||||
`Abort()`, `HasPendingMessages()`, `GetContextUsage()`, `GetSystemPrompt()`, `Compact()`,
|
||||
`PrepareProcessAsync(request)` (see [spawning processes](#spawning-processes)).
|
||||
`PrepareProcessAsync(request)` (see [spawning processes](#spawning-processes)), `Channel` (the host's
|
||||
`IChatChannel`, or null).
|
||||
|
||||
Command handlers get `IExtensionCommandContext`, which adds `WaitForIdleAsync`, `NewSessionAsync` and
|
||||
`SwitchSessionAsync`.
|
||||
|
||||
@@ -1,7 +1,7 @@
|
||||
# Telegram attachments (files in, voice in, files out) — design
|
||||
|
||||
Date: 2026-09-26
|
||||
Status: draft
|
||||
Status: implemented
|
||||
Depends on: `2026-09-26-telegram-message-context-design.md` (`ChatInbox`, `InboundComposer` content steps, lease
|
||||
contract); `2026-09-26-process-environment-hook-design.md` (keeps `OPENAI_API_KEY` out of `bash`). Used by:
|
||||
`2026-09-26-scheduler-design.md` (`IChatChannel.SendTextAsync`).
|
||||
@@ -404,9 +404,10 @@ key.
|
||||
- Cleaning up `inbox/`: files stay until the user or agent removes them.
|
||||
- Reply and forward context headers (the message-context spec).
|
||||
|
||||
## Open questions
|
||||
## Resolved questions
|
||||
|
||||
1. Should `Audio` files (music, podcasts) be transcribed by default, or only saved? This spec follows
|
||||
VolcanicSecretary and transcribes, which costs money on long files.
|
||||
2. Is 3.75 MB the right inline threshold for image documents, or should the host downscale larger images (with a
|
||||
dependency such as ImageSharp) rather than send only the path?
|
||||
1. `Audio` files (music, podcasts) are transcribed by default, as in VolcanicSecretary, and also saved to the inbox.
|
||||
The cost of transcribing long files is accepted.
|
||||
2. 3.75 MB (`Telegram:InlineImageMaxBytes`) stays the inline threshold for image documents, and the host doesn't
|
||||
downscale larger ones: no image-processing dependency. The model gets the path and can downscale with `bash` if it
|
||||
needs to see the image.
|
||||
|
||||
@@ -146,10 +146,7 @@ if the part has a body or images, as in the reference.
|
||||
|
||||
Unsupported kinds never reach the composer: the router checks the message kind before taking a lease. Anything other
|
||||
than text, photo, document, video, audio or voice gets `I can read text, photos, files and voice messages.` (the
|
||||
sibling spec's text, replacing today's "I can read text and photos.").
|
||||
|
||||
Until the attachments spec lands, only text (or a caption) and photos are supported; anything else keeps today's reply
|
||||
`I can read text and photos.` (`InboundComposer.UnsupportedReply`) and takes no lease.
|
||||
sibling spec's text, `InboundComposer.UnsupportedReply`), and takes no lease.
|
||||
|
||||
The sibling spec relies on this contract: **any slow work for a message happens inside `ComposeAsync`, while the
|
||||
router holds that message's lease.** A preparer never talks to `ChatInbox` itself.
|
||||
|
||||
@@ -34,6 +34,11 @@ settings use the `GoogleCalendar` section):
|
||||
| `Telegram:EditInterval` | `00:00:01.5` | Minimum time between edits of the "Working..." status message |
|
||||
| `Telegram:InputDebounce` | `00:00:00.6` | How long the bot waits after a message for more before sending them to the agent as one request. `0` disables the wait; messages still being read together (e.g. a photo still downloading and a text after it) are still sent as one request. Must not be negative |
|
||||
| `Telegram:ReconnectInitialDelay` / `ReconnectMaxDelay` | `1 s` / `60 s` | Backoff after lost connection |
|
||||
| `OPENAI_API_KEY` / `Telegram:Speech:ApiKey` | — | Turns voice transcription on. Hidden from `bash` and redacted from tool output |
|
||||
| `STT_MODEL` / `Telegram:Speech:Model` | `gpt-4o-transcribe` | `gpt-4o-mini-transcribe` and `whisper-1` also work |
|
||||
| `Telegram:Speech:BaseUrl` | `https://api.openai.com/v1` | Any OpenAI-compatible transcription endpoint |
|
||||
| `Telegram:Speech:Language` | — | Optional ISO-639-1 hint, e.g. `de` |
|
||||
| `Telegram:InlineImageMaxBytes` | `3750000` | Image files up to this size are also shown to the model; must be positive |
|
||||
| `GOOGLE_CLIENT_ID` / `GoogleCalendar:ClientId` | — | Optional. With the secret, every chat gets the `calendar` tool and the `/calendar_*` commands; see [google-calendar.md](google-calendar.md) |
|
||||
| `GOOGLE_CLIENT_SECRET` / `GoogleCalendar:ClientSecret` | — | Optional; redacted from tool output and hidden from `bash` |
|
||||
| `PROCESS_ENV_ALLOW` | — | Comma-separated names or globs let through to `bash`, scheduled scripts and MCP servers despite the deny-list (e.g. `GH_TOKEN` for `gh`). Added to `ProcessEnvironment:Allow` |
|
||||
@@ -61,7 +66,7 @@ It lists the last five tool calls (`▶` running, `✓` done, `✗` failed) and
|
||||
Answers are not streamed: each one is sent complete when the model finishes it (split into several messages past
|
||||
Telegram's 4096 character limit).
|
||||
A message sent while the agent works steers the current run, and the bot reacts to it with 👀 so you can see it was
|
||||
added. Photos (with or without a caption) are passed to the model.
|
||||
added. Besides text, you can send photos, files and voice messages (see [What you can send](#what-you-can-send)).
|
||||
|
||||
Along with your text, the agent sees:
|
||||
|
||||
@@ -103,11 +108,33 @@ gets the re-login link and code prompt in their own private chat, regardless of
|
||||
it. That user must already have started a chat with the bot — Telegram bots cannot message a user first — or
|
||||
the login goes unanswered and the triggering request fails ("no code was entered"). The link, the steps and the
|
||||
code prompt arrive as one message. While that prompt is pending, the owner's next plain text message in that chat
|
||||
is taken as the code and is not sent to the agent; photos and `/commands` still behave normally. A forwarded message
|
||||
never counts as the code. Don't ask the bot something else in the owner chat until the code is in: a second free-text
|
||||
is taken as the code and is not sent to the agent; photos, files, voice messages and `/commands` still behave
|
||||
normally. A forwarded message never counts as the code. Don't ask the bot something else in the owner chat until the code is in: a second free-text
|
||||
question in that chat (for example from an extension) replaces the pending prompt and cancels the login. See
|
||||
[claude-code-provider.md](claude-code-provider.md#session-expiry-and-re-login).
|
||||
|
||||
### What you can send
|
||||
|
||||
- **Text and photos.** A photo, with or without a caption, is passed to the model as an image.
|
||||
- **Documents, videos and audio files** are saved to `inbox/<yyyy-MM-dd>-<name>` in the chat's workspace (a number is
|
||||
added when the name is taken: `-1`, `-2`, ...). The agent gets the caption plus a line such as
|
||||
`[File received: inbox/2026-09-26-report.pdf (application/pdf, 1.2 MB)]`. An image sent "as file" (JPEG, PNG, WebP,
|
||||
GIF) up to 3.75 MB (`Telegram:InlineImageMaxBytes`) is also shown to the model; a larger one reaches it as a path
|
||||
only.
|
||||
- **Voice messages and audio files** are transcribed with OpenAI (`gpt-4o-transcribe` by default). The bot echoes
|
||||
`🎤 <transcript>` and the agent gets `[Voice message] <transcript>` (an audio file also gets its `[File received: …]`
|
||||
line). Without an API key the bot replies with how to set one up (`OPENAI_API_KEY`).
|
||||
- Telegram lets bots download at most 20 MB. A bigger file is refused with a note, and nothing reaches the agent.
|
||||
- A message with an attachment is never a command and never answers a pending question: its caption is prompt text.
|
||||
- Anything else (stickers, video notes, locations, polls) gets `I can read text, photos, files and voice messages.`
|
||||
|
||||
### Files from the agent
|
||||
|
||||
The agent's `send` tool delivers a file from the chat's workspace into the chat, before the agent's reply. Images go
|
||||
as photos (up to 10 MB), other files as documents (up to 50 MB), with an optional caption of up to 1024 characters. A
|
||||
photo Telegram rejects (for example for its dimensions) is resent as a document. `send` accepts only files inside the
|
||||
chat's workspace and delivers only to that chat.
|
||||
|
||||
### Google Calendar
|
||||
|
||||
Each chat connects its own Google account with `/calendar_connect`, and the agent's `calendar` tool only uses that
|
||||
@@ -121,6 +148,10 @@ The agent has the built-in tools, including `bash`, and runs them as the bot's O
|
||||
are served (everyone else is ignored and logged), and only in private chats. Run the bot as a dedicated user or in a
|
||||
container, and treat the workspace root as writable by anyone on the allow-list.
|
||||
|
||||
`send` refuses paths outside the chat's workspace, symlinks included, so a prompt-injected "send ~/.ssh/id_rsa" fails
|
||||
visibly (it is no barrier against `bash cp`). The speech API key is kept out of the agent's processes like the bot
|
||||
token.
|
||||
|
||||
Secrets are removed from the environment of `bash`, scheduled scripts and MCP servers: every variable on the deny-list
|
||||
(see `ProcessEnvironment:Deny` above) is dropped, and so is any variable whose value is the bot token, whatever its
|
||||
name (a Coolify alias, for example). `PROCESS_ENV_ALLOW` lets a named variable through, for example `GH_TOKEN` so the
|
||||
|
||||
@@ -5,7 +5,8 @@ The core has no user interface. A UI is a process that
|
||||
1. creates one `AgentSession` per conversation,
|
||||
2. turns incoming user actions into calls on the session (`PromptAsync`, `AbortAsync`, `SetModelAsync`, ...),
|
||||
3. subscribes to `AgentSessionEvent`s and renders them,
|
||||
4. implements `IUserInterface` so extensions can ask the user questions.
|
||||
4. implements `IUserInterface` so extensions can ask the user questions,
|
||||
5. optionally implements `IChatChannel` so the agent can send files (and schedulers text) into the chat.
|
||||
|
||||
This guide uses Telegram as the worked example; the finished bot lives in `hosts/Volcanic.AgenticCore.Telegram`
|
||||
(see [telegram-bot.md](telegram-bot.md)) and is the reference for everything below. Everything transfers to Slack, Discord, a web socket
|
||||
@@ -59,6 +60,8 @@ list) and set `AgenticCoreOptions.SessionStoreFactory`.
|
||||
| Text message while idle | `session.PromptAsync(text, images)` |
|
||||
| Text message while `session.IsStreaming` | `session.PromptAsync(text, images, StreamingBehavior.Steer)` (or `FollowUp`) |
|
||||
| Photo | download it, wrap as `ImageContent(base64, "image/jpeg")`, pass in `images` |
|
||||
| Document / video / audio | download (≤ 20 MB), save into the workspace, mention the path in the prompt text (image documents also as `ImageContent`) |
|
||||
| Voice | transcribe on the host (the core only sees text), put the transcript in the prompt |
|
||||
| `/stop` | `session.AbortAsync()` |
|
||||
| `/model` | list `session.Models.ListModels()` where `IsAvailable`, then `session.SetModelAsync(key)` |
|
||||
| `/thinking <level>` | `session.SetThinkingLevelAsync(level)` (`session.AvailableThinkingLevels`) |
|
||||
@@ -163,7 +166,43 @@ When the user replies to a `ForceReply` prompt, route the text to `TryResolveInp
|
||||
Because `Ui` is not `NullUserInterface`, `ctx.HasUi` is true and extensions like a permission gate will ask
|
||||
instead of blocking.
|
||||
|
||||
## 5. Minimal bot skeleton (Telegram.Bot)
|
||||
## 5. Implement `IChatChannel`
|
||||
|
||||
Optional. `IChatChannel` delivers content into the conversation's chat outside the rendering of assistant
|
||||
messages. Pass one per session as `AgentSessionOptions.Channel`. The core then registers the built-in `send` tool for
|
||||
that session (active by default; an explicit `ActiveTools` still wins), and tools and extensions reach the channel as
|
||||
`ctx.Channel`, host code as `session.Channel`. Sessions without a channel never see `send`.
|
||||
|
||||
```csharp
|
||||
public enum ChatFileKind { Auto, Photo, Document }
|
||||
public sealed record ChatFile(string Path, ChatFileKind Kind = ChatFileKind.Auto, string? Caption = null);
|
||||
public sealed record ChatDelivery(ChatFileKind? SentAs, string? MessageId);
|
||||
public sealed record ChatChannelLimits(long MaxPhotoBytes, long MaxDocumentBytes, int MaxCaptionLength);
|
||||
|
||||
public interface IChatChannel
|
||||
{
|
||||
ChatChannelLimits Limits { get; }
|
||||
Task<ChatDelivery> SendTextAsync(string markdown, CancellationToken cancellationToken);
|
||||
Task<ChatDelivery> SendFileAsync(ChatFile file, CancellationToken cancellationToken);
|
||||
}
|
||||
|
||||
public sealed class ChatDeliveryException(string message, Exception? inner = null) : Exception(message, inner);
|
||||
```
|
||||
|
||||
- `Limits`: your platform's upload limits (Telegram: photo 10 MB, document 50 MB, caption 1024 characters). `send`
|
||||
checks them before calling you, and its errors name them.
|
||||
- `SendTextAsync`: markdown, rendered like assistant text and split when long. It is not added to the session
|
||||
transcript, and it must work while no run is active: schedulers use it for reminders.
|
||||
- `SendFileAsync`: the core has already confined the path to the workspace (symlinks included), checked that the file
|
||||
exists and applied the limits. `Auto` lets you choose, and you may fall back (Telegram resends a rejected photo as a
|
||||
document); report what you did in `SentAs`. Throw only `ChatDeliveryException`, with a message the model or the
|
||||
user can read, or `OperationCanceledException`.
|
||||
- Keep deliveries in order with your rendering: a file sent directly could overtake an assistant message that is
|
||||
still queued for rendering. The Telegram host runs every delivery on its renderer's queue
|
||||
(`ChatRenderer.RunInOrderAsync`), so the file lands after the text that preceded the tool call.
|
||||
- Calls can come from any thread. Nothing delivered is persisted by the core; the `send` tool's result records it.
|
||||
|
||||
## 6. Minimal bot skeleton (Telegram.Bot)
|
||||
|
||||
```csharp
|
||||
var services = new ServiceCollection();
|
||||
@@ -224,7 +263,7 @@ bot.OnUpdate += async (update, _) =>
|
||||
subscribed to; `NewSessionAsync`/`SwitchSessionAsync` keep the same `AgentSession` object, so one
|
||||
subscription per `AgentSession` is enough.
|
||||
|
||||
## 6. Operational notes
|
||||
## 7. Operational notes
|
||||
|
||||
- **Working directories**: the agent's tools (`bash`, `write`, ...) run with the bot process's permissions in
|
||||
the session's `Cwd`. Run the bot in a container or with a dedicated user and one workspace per chat.
|
||||
|
||||
Reference in New Issue
Block a user