diff --git a/CLAUDE.md b/CLAUDE.md index e2ef9e2..c898718 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -45,8 +45,8 @@ Read `docs/architecture.md` first. Key points: - **`src/Volcanic.AgenticCore.Abstractions`** holds the contracts: messages, `Model`/`IProvider`, the streaming event protocol, `ITool`/`Tool`, `IExtension`/`IExtensionApi` and extension events, session entries/stores, - `IUserInterface`, and `AgentSessionEvent`. Extensions reference **only** this assembly, so it stays small and - stable. Don't let runtime types leak into it. + `IUserInterface`, `IChatChannel`, and `AgentSessionEvent`. Extensions reference **only** this assembly, so it stays + small and stable. Don't let runtime types leak into it. - **`src/Volcanic.AgenticCore`** is organised as **vertical slices** under `Features/` (`Agent`, `Sessions`, `Models`, `Tools`, `Extensions`, `Commands`, `SystemPrompt`, `Compaction`, `Processes`). Each slice exposes one entry type (`AgentSession`, `SessionManager`, `ModelRegistry`, `ToolRegistry`, `ExtensionRunner`, `CommandRegistry`, @@ -103,7 +103,10 @@ Long polling (`Polling/`, with `ReconnectPolicy` backoff). Each private chat get limited by `AllowedUserIds`. Incoming messages go through `Inbound/`: the router takes a `ChatInbox` lease per message in update order, `InboundComposer` builds its part (reply/forward headers from `MessageContext`, text and photo) in the background, and the inbox flushes a batch `InputDebounce` (600 ms) after the last message as one prompt -with a local-time header. The first allowed id is the "owner": `OwnerLoginChannel` sends Claude Code re-login +with a local-time header. Files, audio and voice are saved to `/inbox/` (voice transcribed by +`Speech/OpenAiSpeechToText` when `OPENAI_API_KEY` is set) through `InboundComposer`'s content steps +(`Inbound/InboundMedia`), and `Chats/TelegramChatChannel` implements `IChatChannel` for the core `send` tool through +`ChatRenderer.RunInOrderAsync`. The first allowed id is the "owner": `OwnerLoginChannel` sends Claude Code re-login prompts to that chat. It resolves lazily to break the DI cycle ChatRegistry → session factory → extension → ChatRegistry. Answers go out complete, not streamed (`Rendering/`). Tests drive the host through `Support/FakeBotClient` and `ChatHost`. diff --git a/docs/architecture.md b/docs/architecture.md index 2b8ceb5..79d0d8c 100644 --- a/docs/architecture.md +++ b/docs/architecture.md @@ -12,7 +12,8 @@ tests/* xunit `Abstractions` is deliberately small and stable: messages and content blocks, `Model` and `IProvider`, the streaming event protocol, `ITool`, the extension API and its events, session entries and stores, -`IUserInterface`, the `AgentSessionEvent` family a UI subscribes to, and the process contract in `Processes/` +`IUserInterface`, `IChatChannel` (delivering files and texts into the conversation's chat), the `AgentSessionEvent` +family a UI subscribes to, and the process contract in `Processes/` (`IProcessSpawnPolicy`, `ProcessSpawnRequest`/`ProcessSpawnContext`, `ProcessSpawnHook`, `ProcessSpawnEvent`, `ISecretRedactor`). Extensions reference only this assembly, which is what allows them to be loaded from separate directories. @@ -29,7 +30,7 @@ owns everything for that feature and exposes one entry type: |---|---|---| | `Models` | `ModelRegistry` | Provider registrations, `provider/id` lookup, API key resolution, transcript normalisation before a call | | `Sessions` | `SessionManager` | The entry tree of one session, branch/context building, `ISessionStore` implementations | -| `Tools` | `ToolRegistry` | Known and active tools, argument validation, output truncation, the built-in tools | +| `Tools` | `ToolRegistry` | Known and active tools, argument validation, output truncation, the built-in tools (`send` only for sessions with a channel) | | `Extensions` | `ExtensionRunner` | Activating extensions (two-phase), dispatching events, contexts, the event bus, assembly loading | | `Commands` | `CommandRegistry` | Slash commands | | `SystemPrompt` | `SystemPromptBuilder` | Base prompt, tools section, guidelines, project context files, environment | diff --git a/docs/extensions.md b/docs/extensions.md index d575c0d..34345cb 100644 --- a/docs/extensions.md +++ b/docs/extensions.md @@ -115,7 +115,8 @@ fails the spawn); `Cancel`, `Block` and `MarkHandled` stop further handlers. status), `SessionManager` (read-only entries and branch), `ModelRegistry` (models and provider-neutral `CompleteAsync` for nested model calls), `Model`, `ThinkingLevel`, `Signal` (cancelled on abort), `IsIdle()`, `Abort()`, `HasPendingMessages()`, `GetContextUsage()`, `GetSystemPrompt()`, `Compact()`, -`PrepareProcessAsync(request)` (see [spawning processes](#spawning-processes)). +`PrepareProcessAsync(request)` (see [spawning processes](#spawning-processes)), `Channel` (the host's +`IChatChannel`, or null). Command handlers get `IExtensionCommandContext`, which adds `WaitForIdleAsync`, `NewSessionAsync` and `SwitchSessionAsync`. diff --git a/docs/superpowers/specs/2026-09-26-telegram-attachments-design.md b/docs/superpowers/specs/2026-09-26-telegram-attachments-design.md index 091c3fd..b3f49be 100644 --- a/docs/superpowers/specs/2026-09-26-telegram-attachments-design.md +++ b/docs/superpowers/specs/2026-09-26-telegram-attachments-design.md @@ -1,7 +1,7 @@ # Telegram attachments (files in, voice in, files out) — design Date: 2026-09-26 -Status: draft +Status: implemented Depends on: `2026-09-26-telegram-message-context-design.md` (`ChatInbox`, `InboundComposer` content steps, lease contract); `2026-09-26-process-environment-hook-design.md` (keeps `OPENAI_API_KEY` out of `bash`). Used by: `2026-09-26-scheduler-design.md` (`IChatChannel.SendTextAsync`). @@ -404,9 +404,10 @@ key. - Cleaning up `inbox/`: files stay until the user or agent removes them. - Reply and forward context headers (the message-context spec). -## Open questions +## Resolved questions -1. Should `Audio` files (music, podcasts) be transcribed by default, or only saved? This spec follows - VolcanicSecretary and transcribes, which costs money on long files. -2. Is 3.75 MB the right inline threshold for image documents, or should the host downscale larger images (with a - dependency such as ImageSharp) rather than send only the path? +1. `Audio` files (music, podcasts) are transcribed by default, as in VolcanicSecretary, and also saved to the inbox. + The cost of transcribing long files is accepted. +2. 3.75 MB (`Telegram:InlineImageMaxBytes`) stays the inline threshold for image documents, and the host doesn't + downscale larger ones: no image-processing dependency. The model gets the path and can downscale with `bash` if it + needs to see the image. diff --git a/docs/superpowers/specs/2026-09-26-telegram-message-context-design.md b/docs/superpowers/specs/2026-09-26-telegram-message-context-design.md index 9112314..56c2a3f 100644 --- a/docs/superpowers/specs/2026-09-26-telegram-message-context-design.md +++ b/docs/superpowers/specs/2026-09-26-telegram-message-context-design.md @@ -146,10 +146,7 @@ if the part has a body or images, as in the reference. Unsupported kinds never reach the composer: the router checks the message kind before taking a lease. Anything other than text, photo, document, video, audio or voice gets `I can read text, photos, files and voice messages.` (the -sibling spec's text, replacing today's "I can read text and photos."). - -Until the attachments spec lands, only text (or a caption) and photos are supported; anything else keeps today's reply -`I can read text and photos.` (`InboundComposer.UnsupportedReply`) and takes no lease. +sibling spec's text, `InboundComposer.UnsupportedReply`), and takes no lease. The sibling spec relies on this contract: **any slow work for a message happens inside `ComposeAsync`, while the router holds that message's lease.** A preparer never talks to `ChatInbox` itself. diff --git a/docs/telegram-bot.md b/docs/telegram-bot.md index d91221c..8e6caf1 100644 --- a/docs/telegram-bot.md +++ b/docs/telegram-bot.md @@ -34,6 +34,11 @@ settings use the `GoogleCalendar` section): | `Telegram:EditInterval` | `00:00:01.5` | Minimum time between edits of the "Working..." status message | | `Telegram:InputDebounce` | `00:00:00.6` | How long the bot waits after a message for more before sending them to the agent as one request. `0` disables the wait; messages still being read together (e.g. a photo still downloading and a text after it) are still sent as one request. Must not be negative | | `Telegram:ReconnectInitialDelay` / `ReconnectMaxDelay` | `1 s` / `60 s` | Backoff after lost connection | +| `OPENAI_API_KEY` / `Telegram:Speech:ApiKey` | — | Turns voice transcription on. Hidden from `bash` and redacted from tool output | +| `STT_MODEL` / `Telegram:Speech:Model` | `gpt-4o-transcribe` | `gpt-4o-mini-transcribe` and `whisper-1` also work | +| `Telegram:Speech:BaseUrl` | `https://api.openai.com/v1` | Any OpenAI-compatible transcription endpoint | +| `Telegram:Speech:Language` | — | Optional ISO-639-1 hint, e.g. `de` | +| `Telegram:InlineImageMaxBytes` | `3750000` | Image files up to this size are also shown to the model; must be positive | | `GOOGLE_CLIENT_ID` / `GoogleCalendar:ClientId` | — | Optional. With the secret, every chat gets the `calendar` tool and the `/calendar_*` commands; see [google-calendar.md](google-calendar.md) | | `GOOGLE_CLIENT_SECRET` / `GoogleCalendar:ClientSecret` | — | Optional; redacted from tool output and hidden from `bash` | | `PROCESS_ENV_ALLOW` | — | Comma-separated names or globs let through to `bash`, scheduled scripts and MCP servers despite the deny-list (e.g. `GH_TOKEN` for `gh`). Added to `ProcessEnvironment:Allow` | @@ -61,7 +66,7 @@ It lists the last five tool calls (`▶` running, `✓` done, `✗` failed) and Answers are not streamed: each one is sent complete when the model finishes it (split into several messages past Telegram's 4096 character limit). A message sent while the agent works steers the current run, and the bot reacts to it with 👀 so you can see it was -added. Photos (with or without a caption) are passed to the model. +added. Besides text, you can send photos, files and voice messages (see [What you can send](#what-you-can-send)). Along with your text, the agent sees: @@ -103,11 +108,33 @@ gets the re-login link and code prompt in their own private chat, regardless of it. That user must already have started a chat with the bot — Telegram bots cannot message a user first — or the login goes unanswered and the triggering request fails ("no code was entered"). The link, the steps and the code prompt arrive as one message. While that prompt is pending, the owner's next plain text message in that chat -is taken as the code and is not sent to the agent; photos and `/commands` still behave normally. A forwarded message -never counts as the code. Don't ask the bot something else in the owner chat until the code is in: a second free-text +is taken as the code and is not sent to the agent; photos, files, voice messages and `/commands` still behave +normally. A forwarded message never counts as the code. Don't ask the bot something else in the owner chat until the code is in: a second free-text question in that chat (for example from an extension) replaces the pending prompt and cancels the login. See [claude-code-provider.md](claude-code-provider.md#session-expiry-and-re-login). +### What you can send + +- **Text and photos.** A photo, with or without a caption, is passed to the model as an image. +- **Documents, videos and audio files** are saved to `inbox/-` in the chat's workspace (a number is + added when the name is taken: `-1`, `-2`, ...). The agent gets the caption plus a line such as + `[File received: inbox/2026-09-26-report.pdf (application/pdf, 1.2 MB)]`. An image sent "as file" (JPEG, PNG, WebP, + GIF) up to 3.75 MB (`Telegram:InlineImageMaxBytes`) is also shown to the model; a larger one reaches it as a path + only. +- **Voice messages and audio files** are transcribed with OpenAI (`gpt-4o-transcribe` by default). The bot echoes + `🎤 ` and the agent gets `[Voice message] ` (an audio file also gets its `[File received: …]` + line). Without an API key the bot replies with how to set one up (`OPENAI_API_KEY`). +- Telegram lets bots download at most 20 MB. A bigger file is refused with a note, and nothing reaches the agent. +- A message with an attachment is never a command and never answers a pending question: its caption is prompt text. +- Anything else (stickers, video notes, locations, polls) gets `I can read text, photos, files and voice messages.` + +### Files from the agent + +The agent's `send` tool delivers a file from the chat's workspace into the chat, before the agent's reply. Images go +as photos (up to 10 MB), other files as documents (up to 50 MB), with an optional caption of up to 1024 characters. A +photo Telegram rejects (for example for its dimensions) is resent as a document. `send` accepts only files inside the +chat's workspace and delivers only to that chat. + ### Google Calendar Each chat connects its own Google account with `/calendar_connect`, and the agent's `calendar` tool only uses that @@ -121,6 +148,10 @@ The agent has the built-in tools, including `bash`, and runs them as the bot's O are served (everyone else is ignored and logged), and only in private chats. Run the bot as a dedicated user or in a container, and treat the workspace root as writable by anyone on the allow-list. +`send` refuses paths outside the chat's workspace, symlinks included, so a prompt-injected "send ~/.ssh/id_rsa" fails +visibly (it is no barrier against `bash cp`). The speech API key is kept out of the agent's processes like the bot +token. + Secrets are removed from the environment of `bash`, scheduled scripts and MCP servers: every variable on the deny-list (see `ProcessEnvironment:Deny` above) is dropped, and so is any variable whose value is the bot token, whatever its name (a Coolify alias, for example). `PROCESS_ENV_ALLOW` lets a named variable through, for example `GH_TOKEN` so the diff --git a/docs/ui-integration.md b/docs/ui-integration.md index 70f8419..1a34e94 100644 --- a/docs/ui-integration.md +++ b/docs/ui-integration.md @@ -5,7 +5,8 @@ The core has no user interface. A UI is a process that 1. creates one `AgentSession` per conversation, 2. turns incoming user actions into calls on the session (`PromptAsync`, `AbortAsync`, `SetModelAsync`, ...), 3. subscribes to `AgentSessionEvent`s and renders them, -4. implements `IUserInterface` so extensions can ask the user questions. +4. implements `IUserInterface` so extensions can ask the user questions, +5. optionally implements `IChatChannel` so the agent can send files (and schedulers text) into the chat. This guide uses Telegram as the worked example; the finished bot lives in `hosts/Volcanic.AgenticCore.Telegram` (see [telegram-bot.md](telegram-bot.md)) and is the reference for everything below. Everything transfers to Slack, Discord, a web socket @@ -59,6 +60,8 @@ list) and set `AgenticCoreOptions.SessionStoreFactory`. | Text message while idle | `session.PromptAsync(text, images)` | | Text message while `session.IsStreaming` | `session.PromptAsync(text, images, StreamingBehavior.Steer)` (or `FollowUp`) | | Photo | download it, wrap as `ImageContent(base64, "image/jpeg")`, pass in `images` | +| Document / video / audio | download (≤ 20 MB), save into the workspace, mention the path in the prompt text (image documents also as `ImageContent`) | +| Voice | transcribe on the host (the core only sees text), put the transcript in the prompt | | `/stop` | `session.AbortAsync()` | | `/model` | list `session.Models.ListModels()` where `IsAvailable`, then `session.SetModelAsync(key)` | | `/thinking ` | `session.SetThinkingLevelAsync(level)` (`session.AvailableThinkingLevels`) | @@ -163,7 +166,43 @@ When the user replies to a `ForceReply` prompt, route the text to `TryResolveInp Because `Ui` is not `NullUserInterface`, `ctx.HasUi` is true and extensions like a permission gate will ask instead of blocking. -## 5. Minimal bot skeleton (Telegram.Bot) +## 5. Implement `IChatChannel` + +Optional. `IChatChannel` delivers content into the conversation's chat outside the rendering of assistant +messages. Pass one per session as `AgentSessionOptions.Channel`. The core then registers the built-in `send` tool for +that session (active by default; an explicit `ActiveTools` still wins), and tools and extensions reach the channel as +`ctx.Channel`, host code as `session.Channel`. Sessions without a channel never see `send`. + +```csharp +public enum ChatFileKind { Auto, Photo, Document } +public sealed record ChatFile(string Path, ChatFileKind Kind = ChatFileKind.Auto, string? Caption = null); +public sealed record ChatDelivery(ChatFileKind? SentAs, string? MessageId); +public sealed record ChatChannelLimits(long MaxPhotoBytes, long MaxDocumentBytes, int MaxCaptionLength); + +public interface IChatChannel +{ + ChatChannelLimits Limits { get; } + Task SendTextAsync(string markdown, CancellationToken cancellationToken); + Task SendFileAsync(ChatFile file, CancellationToken cancellationToken); +} + +public sealed class ChatDeliveryException(string message, Exception? inner = null) : Exception(message, inner); +``` + +- `Limits`: your platform's upload limits (Telegram: photo 10 MB, document 50 MB, caption 1024 characters). `send` + checks them before calling you, and its errors name them. +- `SendTextAsync`: markdown, rendered like assistant text and split when long. It is not added to the session + transcript, and it must work while no run is active: schedulers use it for reminders. +- `SendFileAsync`: the core has already confined the path to the workspace (symlinks included), checked that the file + exists and applied the limits. `Auto` lets you choose, and you may fall back (Telegram resends a rejected photo as a + document); report what you did in `SentAs`. Throw only `ChatDeliveryException`, with a message the model or the + user can read, or `OperationCanceledException`. +- Keep deliveries in order with your rendering: a file sent directly could overtake an assistant message that is + still queued for rendering. The Telegram host runs every delivery on its renderer's queue + (`ChatRenderer.RunInOrderAsync`), so the file lands after the text that preceded the tool call. +- Calls can come from any thread. Nothing delivered is persisted by the core; the `send` tool's result records it. + +## 6. Minimal bot skeleton (Telegram.Bot) ```csharp var services = new ServiceCollection(); @@ -224,7 +263,7 @@ bot.OnUpdate += async (update, _) => subscribed to; `NewSessionAsync`/`SwitchSessionAsync` keep the same `AgentSession` object, so one subscription per `AgentSession` is enough. -## 6. Operational notes +## 7. Operational notes - **Working directories**: the agent's tools (`bash`, `write`, ...) run with the bot process's permissions in the session's `Cwd`. Run the bot in a container or with a dedicated user and one workspace per chat.