Externe Integrationen: Möglichkeit zur Erstellung von Integrationen des Typs „TTS-Anbieter“

pierre-gilles · 1. August 2026 um 10:43

Externe TTS-Integrationen wie „Gradium TTS“, „Whisper“ usw. erstellen können.

pierre-gilles · 3. August 2026 um 14:52

Der PR auf der Seite von Gladys:

github.com/GladysAssistant/Gladys

Add TTS provider system with external integration support (#2746)

master ← claude/external-tts-provider-integration-07klnl

ouvert 07:35AM - 02 Aug 26 UTC

Pierre-Gilles

+1525 -55

### Description This PR implements a pluggable text-to-speech (TTS) provider sy…stem that decouples TTS consumers (scene actions, voice assistant) from specific TTS engines. The system supports: 1. **Built-in Gladys Plus provider** as the default, with fallback behavior 2. **External TTS integrations** (type: "tts") that expose a `synthesize` command over WebSocket 3. **Provider selection UI** on the Artificial Intelligence integration page to switch between available providers 4. **Audio streaming** via high-entropy tokens for LAN speakers (Sonos, Cast, etc.) that cannot send JWT authentication #### Key Changes **Backend:** - New `Tts` manager class (`server/lib/tts/`) with methods to: - List available providers (`getProviders`) - Get/set active provider configuration (`getProviderConfiguration`, `setActiveProvider`) - Generate speech URLs (`getSpeechUrl`) - dispatches to Gladys Plus or external provider - Serve synthesized audio clips (`getAudio`) with token-based access control - Manage audio lifecycle (purge expired clips, get local API base URL) - TTS controller with three endpoints: - `GET /api/v1/tts/provider` - list providers and active selection - `POST /api/v1/tts/provider` - change active provider (admin only) - `GET /api/v1/tts/audio/:token` - serve audio clips (unauthenticated) - Extended external integration proxy service registration to expose `tts.synthesize` capability for TTS-type integrations - Validation of TTS manifest (rejects messaging/contact_schema fields) - Audio validation: content-type curation, size bounds, text length capping **Frontend:** - New `TtsProviderSettings` component for provider selection on the Artificial Intelligence integration page (`front/src/routes/integration/all/openai/`), next to the other voice/chat features - Updated `PlayNotification` scene action to conditionally show Gladys Plus upsell only when using Gladys Plus - Integration catalog support for TTS providers (empty by default, populated by external integrations) - i18n strings for TTS category in English, French, and German **Tests:** - Comprehensive test suite for TTS manager (providers, configuration, speech URL generation, audio retrieval) - External integration TTS provider tests (manifest validation, proxy service registration, audio handling) - TTS controller tests (provider endpoints, audio serving) - Updated existing tests to use new TTS system #### Design Notes - Audio clips stored in RAM only (no database table) behind high-entropy tokens - the token is the secret - 10-minute TTL for synthesized clips with lazy purging - Text capped before relay to provider (truncation, not error) - Unauthenticated audio endpoint by design: speakers cannot send JWT, token provides security - Local API base URL derived from the LAN-facing IPv4 (Docker bridges excluded, physical NICs preferred over VPN tunnels) + SERVER_PORT for LAN accessibility - Graceful fallback: if configured provider is unavailable, throws rather than silently switching ### Checklist - [x] Tests pass: Added comprehensive unit tests for TTS manager, external integration TTS support, and controller endpoints - [x] Linter and prettier pass - [x] No breaking changes: New system is additive; existing Gladys Plus behavior unchanged when no provider configured https://claude.ai/code/session_013ihJhVk52zW6qUsxBHeqx2 ## Summary by CodeRabbit * **New Features** * Added text-to-speech provider support, including provider selection and configuration. * Added a dedicated TTS integration category with installation guidance and multilingual translations. * Enabled TTS audio for scene announcements and voice-assistant responses. * Added local-network audio delivery with short-lived secure URLs. * Falls back to Gladys Plus when no external provider is selected or available. * **Bug Fixes** * Removing the active TTS provider now automatically restores Gladys Plus. * Added validation for unsupported, oversized, malformed, or unavailable audio responses. ## Summary by CodeRabbit * **New Features** * Added text-to-speech integrations with provider selection and configuration. * Added support for scene announcements and voice-assistant responses using the selected provider. * Added local-network audio delivery with Gladys Plus fallback. * Added a dedicated TTS integration category and administrator settings. * Added English, French, and German localization. * **Bug Fixes** * Automatically restores Gladys Plus when the active TTS provider is removed. * Added validation and clearer handling for unavailable or invalid audio responses.

Der PR auf der Seite des SDKs:

github.com/GladysAssistant/integration-sdk-js

feat: support the TTS provider integration type (B.20) (#21)

main ← claude/sdk-changes-pr-2746-lmaqxo

ouvert 03:46PM - 03 Aug 26 UTC

Pierre-Gilles

+257 -15

SDK counterpart of GladysAssistant/Gladys#2746 (B.20 "TTS provider" type): integ…rations declaring `type: "tts"` in their manifest become selectable as the voice of the instance, and answer synthesis requests over the existing WebSocket command channel. ## What's added - **`WEBSOCKET_MESSAGE_TYPES.EXTERNAL_INTEGRATION.TTS_SYNTHESIZE`** (`external-integration.tts.synthesize`), routed in `_handleMessage` like the other auto-acked commands. - **`gladys.onTtsSynthesize(cb)`** — `(text, language) => Promise<string>` resolving the audio data-URI (`<content_type>;base64,...`), acked back as `data.audio` (contract C.8). `language` is normalized to `null` when absent (best-effort hint: the user's language for the voice assistant, `null` in scenes). - **SDK-side validation of the resolved audio** (`_mapTtsAudio`, same pattern as the sync-webhook response mapping): curated content types (`audio/mpeg`, `audio/wav`, `audio/ogg`, `audio/aac`), decoded audio of 1 byte to 5 MB — the same bounds the core enforces in `externalIntegration.registerProxyService.js`, so a bad audio fails the ack with an explicit message on the integration side instead of an opaque `EXTERNAL_INTEGRATION_INVALID_TTS_AUDIO` on the Gladys side. - **Typings** (`onTtsSynthesize`, `TTS_SYNTHESIZE`) + type-test coverage. - **Tests** (`test/tts.test.js`, 10 cases): success ack with `data.audio`, `language` defaulting to `null`, handler throw, missing handler ("not implemented"), non-string / missing separator / non-curated content type / empty audio / > 5 MB rejections, exactly-5 MB accepted. - **README**: handler-table row + a "TTS provider (the voice of the instance)" section, mirroring the camera/communication sections. ## Checks - `npm test`: 190/190 pass - `npm run lint`, `npm run check-types`, `npm run prettier-check`: clean 🤖 Generated with [Claude Code](https://claude.com/claude-code) https://claude.ai/code/session_01LCCieF9vzy56ULFi9r6b2g --- _Generated by [Claude Code](https://claude.ai/code/session_01LCCieF9vzy56ULFi9r6b2g)_ ## Summary by CodeRabbit - **New Features** - Added text-to-speech synthesis support for integrations. - Handlers can receive text with an optional language hint and return playable base64 audio. - Supports MP3, WAV, OGG, and AAC audio up to 5 MB. - Added validation for audio formats, data, and response limits. - **Documentation** - Documented TTS provider setup, configuration, playback behavior, input limits, and validation requirements. - Added SDK API documentation for registering TTS synthesis handlers.

Thema		Antworten	Aufrufe
Service Gradium AI Développement	5	129	4. August 2026
Intégrations externes : pouvoir créer des intégrations de type "fournisseur IA" Demande de fonctionnalités in-progress , feature-core	1	50	3. August 2026
Pouvoir parler avec l'IA dans la page de discussion Demandes livrées feature-core , in-progress	2	63	17. August 2026
Problème installation intégration Freebox Configuration integration-externe	65	452	16. August 2026
Gladys Assistant 4.77 : Un assistant vocal dans Gladys 🎙️ Actualités	8	205	15. Juni 2026

Externe Integrationen: Möglichkeit zur Erstellung von Integrationen des Typs „TTS-Anbieter“

Verwandte Themen