feat(telegram): ElevenLabs text to speech for the voice tool

ElevenLabs speaks for the voice tool when ELEVENLABS_API_KEY and ELEVENLABS_VOICE_ID
are set (TTS_PROVIDER=elevenlabs|openai picks, the other is the fallback). Opus output
(opus_48000_64) goes to Telegram as is; voice names resolve through /v2/voices;
instructions become audio tags on eleven_v3 and are ignored on other models. The key
is redacted from tool output and never reaches agent processes.
This commit is contained in:
EugeneTes
2026-10-14 13:07:36 +00:00
parent 6e3241fed9
commit 7fef3ce94e
17 changed files with 554 additions and 21 deletions

View File

@@ -48,7 +48,12 @@ settings use the `GoogleCalendar` section, the scheduler the `Scheduler` section
| `STT_MODEL` / `Telegram:Speech:Model` | `gpt-4o-transcribe` | `gpt-4o-mini-transcribe` and `whisper-1` also work |
| `TTS_MODEL` / `Telegram:Speech:TtsModel` | `gpt-4o-mini-tts` | Speech model of the `voice` tool; `tts-1` and `tts-1-hd` also work (they ignore `instructions`) |
| `TTS_VOICE` / `Telegram:Speech:TtsVoice` | `nova` | Default voice of the `voice` tool |
| `Telegram:Speech:TtsInstructions` | — | Default delivery instructions (tone, pace) for the `voice` tool |
| `Telegram:Speech:TtsInstructions` | — | Default delivery instructions (tone, pace) for the `voice` tool (OpenAI only) |
| `TTS_PROVIDER` / `Telegram:Speech:TtsProvider` | — | `elevenlabs` or `openai`. Empty prefers ElevenLabs. The preferred service is used when configured, else the other |
| `ELEVENLABS_API_KEY` / `Telegram:Speech:ElevenLabs:ApiKey` | — | With a voice id, turns on ElevenLabs for the `voice` tool. Hidden from `bash` and redacted from tool output |
| `ELEVENLABS_VOICE_ID` / `Telegram:Speech:ElevenLabs:VoiceId` | — | Default ElevenLabs voice: a voice id, or the name of a voice in the account |
| `ELEVENLABS_MODEL` / `Telegram:Speech:ElevenLabs:Model` | `eleven_multilingual_v2` | ElevenLabs model; `eleven_v3` reads `instructions` as audio tags, other models ignore them |
| `Telegram:Speech:ElevenLabs:OutputFormat` | `opus_48000_64` | Must be an `opus_48000_*` format (OGG/Opus) |
| `Telegram:Speech:BaseUrl` | `https://api.openai.com/v1` | Any OpenAI-compatible audio endpoint (transcription and speech) |
| `Telegram:Speech:Language` | — | Optional ISO-639-1 hint, e.g. `de` |
| `OPENROUTER_API_KEY` / `Telegram:Triggers:ApiKey` | — | Turns on [groups and triggers](#groups-and-triggers). Hidden from `bash` and redacted from tool output |
@@ -170,6 +175,12 @@ question in that chat (for example from an extension) replaces the pending promp
text over 1500 characters is split at paragraph, sentence or word boundaries into several voice messages. Longer
text is refused. Every part is synthesized before the first is sent, so an API error (returned to the agent as text,
key hidden) never leaves half a message; synthesis is retried like transcription.
**ElevenLabs** takes over when `ELEVENLABS_API_KEY` and `ELEVENLABS_VOICE_ID` are set (and `TTS_PROVIDER` isn't
`openai`); it needs no OpenAI key. It calls `POST /v1/text-to-speech/{voice_id}?output_format=opus_48000_64` with
`eleven_multilingual_v2` by default, so the audio is OGG/Opus already. The tool's `voice` is a voice id or the name
of a voice in the account (looked up once through `/v2/voices`); `instructions` become audio tags (`[warm]`) on
`eleven_v3` and are ignored on other models. Errors such as `HTTP 401 quota_exceeded: …` or `HTTP 429 …` come back
as text; 429 and 5xx are retried. The tool description follows the active service. The startup log line names it.
- Telegram lets bots download at most 20 MB. A bigger file is refused with a note, and nothing reaches the agent.
- A message with an attachment is never a command and never answers a pending question: its caption is prompt text.
- Anything else (stickers, video notes, locations, polls) gets `I can read text, photos, files and voice messages.`