Support for MediaPlayback, KeypadInput, and LevelControl clusters in media devices

Feature requests
1

Hello everyone!

I just tested the new Matterbridge AI Plugin Factory by Pierre-Gilles to integrate my Freebox Player Delta (Devialet) into Gladys via Matter.

The generated plugin works very well on the Matterbridge side:
:white_check_mark: Authentication via validation on the Freebox screen
:white_check_mark: Automatic discovery of the player
:white_check_mark: Device visible in Gladys via Matter

The plugin correctly exposes the following Matter clusters:

  • OnOff
  • LevelControl (volume)
  • MediaPlayback (play/pause/stop/next/prev)
  • KeypadInput (virtual remote)

But on the Gladys side, only the OnOff cluster is recognized (as a « switch »), and it doesn’t work either because the on/off of a media player is different from a light bulb.

Is it possible to add support for the following clusters in the Gladys Matter integration?

  • MediaPlayback → play, pause, stop, next, previous
  • KeypadInput → sending keys (ok, back, home, up, down, left, right)
  • LevelControl in a media context → volume (not brightness)
  • OnOff for a media player → power on/off (different from a lamp)

This would be a huge step forward for all users who have media devices (TVs, players, TV boxes…) and use Matterbridge to integrate them.

The Freebox plugin is available here for those who want to test it:

Thanks!

2

Yes, that’s absolutely possible! And that’s exactly the idea behind Matter. I want to cover everything at 100% in Gladys so that all non-Matter integrations can then be done in Matterbridge.

I’ll look into it as soon as I can. Otherwise, if anyone wants to make a PR, they’re welcome to do so :blush:

4

Update — PR submitted + mapping of missing Matter clusters

Hello everyone!

A quick update on my Matter/Matterbridge adventure,
in all transparency: I’m not a developer at all.
I rely on Claude (Anthropic’s IA) and the Cursor editor
to help me understand the code and contribute.
Without them, I would never have been able to do what follows :slightly_smiling_face:

What’s happened since my initial post

Following Pierre-Gilles’ feedback that a PR would
be welcome, we got started. Here’s the result:

:backhand_index_pointing_right: PR #2533 submitted to the Gladys repo:

It adds support for three missing Matter clusters:

  • MediaPlayback → play, pause, stop
  • KeypadInput → navigation (up/down/left/right/ok/back)
  • LevelControl (without lighting) → volume

All this so that media devices bridged via Matterbridge
(TV, box, players…) expose real features
in Gladys, not just a switch.

The CodeRabbit bot has already done an automatic review with
two minor nitpicks that I’m currently fixing.

Mapping of missing Matter clusters

While digging into the subject, I compared
all the features available in MQTT and their coverage
in Matter. Result: about 50% of Gladys features
still don’t have a Matter cluster.

High priorities (standard Matter clusters, well documented):

  • DoorLock → LOCK (locks)
  • PowerSource → BATTERY (battery level)
  • SmokeCoAlarm → SMOKE_SENSOR + CO_SENSOR
  • BooleanState → OPENING_SENSOR (opening sensors)

Medium priorities:

  • ColorControl temperature → white color (warm/cold)
  • FanControl → ventilation/air conditioning mode
  • CarbonDioxideConcentration → CO2_SENSOR

Features that don’t exist anywhere yet :light_bulb:
Thinking further, there are uses that neither MQTT
nor Matter cover today in Gladys:

  • Valve/irrigation (the Matter ValveConfigAndControl
    cluster already exists!)
  • Water/gas meter (consumption in m³)
  • Tank level (fuel tank, pool)
  • Human presence radar mmWave (more accurate than a PIR)

Question for the community

Are there features missing from Gladys
today? Devices you can’t
integrate properly?

And if real developers want to take over these
priority clusters, the path is now marked out —
the two files to modify are
convertToGladysDevice.js and matter.setValue.js,
and the pattern is very clear to follow :slightly_smiling_face:

5

Hi @b3n.0 :slight_smile:

I don’t know if you saw in the PR, but we run dozens of checks to ensure the PR is good: unit tests, linting, formatting, code coverage, etc…

You also need to test the change in real conditions before being able to validate it. I don’t know if you’ve already been able to do this part?

You will find in the PR a checklist to complete before moving on to the next step. I invite you to go through the different points and let me know once the fixes are applied and the checks are done.

If it’s too complicated for you, let me know and I’ll make another PR :slight_smile:

6

Hi @pierre-gilles :slightly_smiling_face:

Thanks for your feedback, I’ve gone through the checklist and applied the requested fixes using Cursor.

Real-world testing: not yet, I need to dig through a closet to find an old Raspberry, I think I have an old RPi 2B, hoping it will work! ?

7

Great, but you should be careful as your PR modifies files outside the scope of your PR (Scene.cy.js)

It’s too old, we no longer do armv6 builds ^^

You can test on your dev machine though, no need to necessarily run it remotely :slight_smile:

8

The PR has been closed and a new one created by limiting the scope of the PR…

The tests have been passed, missing the test in real conditions:
npm run dev fails (apparently Node.js too recent on my machine… Is that possible?)

9

Thanks for your PR, but I have a small question. You chose to use a « Television » feature in Gladys. Is MediaPlayback in Matter necessarily linked to a television, or is it also used for music?

How is this cluster displayed on other Matter systems? On iOS, for example?

10

And yes …! In the Matter spec, MediaPlayback is a generic cluster — it’s not reserved for TVs. It is used by TV boxes, smart speakers, music players, etc.

The Matter spec distinguishes two main device types:

  • « Casting Video Player » (TV, box)
  • « Basic Audio Player » / « Speaker » (speakers, audio players)

…but both can use MediaPlayback. Then, depending on the final use, Channel, KeypadInput, and LevelControl are added.

For display in iOS Home, honestly, I don’t know — it’s a good question and I haven’t had the opportunity to test.

So your question about design choice:
Is it better to stick with the existing TELEVISION category in Gladys
(which already has all the play/pause/volume/navigation types), or use MUSIC, or create something more generic?

I completely leave that up to your judgment :slightly_smiling_face:

11

Hi @b3n.0,

It’s really cool that you want to help, and don’t take my feedback the wrong way :smiley:

The thing is, if you just pass my messages to Claude and send me back his response, I could do the same thing directly on my side, so the intermediary loses some of its appeal :joy:

What I’m really looking for is the part that Claude can’t do for you: answering the questions I asked you! That’s the job of a human who knows their context, and that’s where you add real value :smiley:

12

Hi @pierre-gilles,

Of course, I don’t take it personally. I know I’m limited in helping with these kinds of topics :slightly_smiling_face:

I started this PR mainly to test the plugins I requested from the matterbridge-ai-plugin-factory, especially for the Freebox Player and the Yoto of my children. I thought it might also benefit the project along the way.

In both cases, we’re talking about features not covered by Matter in Gladys today.

So, I tried to figure it out with Cursor. But indeed, I still lack a lot of knowledge, even just the vocabulary. I always try to ask the AI for the most didactic explanations possible, but it doesn’t cover everything: I’m still far from mastering the subject.

And if, from your point of view, it ultimately requires more work for you to review than to make the changes yourself, I completely understand. In my job as an architect, I often spend more time explaining to someone how to draw a detail than drawing it myself :slightly_smiling_face:

If for you it’s more work to review than to do it yourself, I understand. In my job as an architect, I sometimes spend more time explaining how to draw a detail than drawing it myself!

13

It’s up to you how you feel about it :slightly_smiling_face:

Frankly, with Cursor, even if you’re not a professional developer, I think it’s possible to create very good PRs for targeted features.

However, this doesn’t eliminate all the upstream work: looking at what the market leaders are doing (this avoids reinventing the wheel), thinking about the user experience, and then testing in practice that it works well and is pleasant to use. The AI doesn’t really do these parts for you, that’s where your contribution is most important.

If you have the motivation to carry out a development from start to finish, from the idea to the tests, it helps me enormously. And in addition, you will get the features you are interested in in Gladys much faster :grinning_face_with_smiling_eyes:

14

And yes, that was really the basis of my motivation.

Now I understand that I need to be able to complete the tests. On that, I still have some progress to make…

15

Hi @pierre-gilles,

I’ve looked around a bit, and it’s mainly at HA that we have leads…

So, I see three possible approaches:

  1. A single widget that automatically adapts to the device’s features (channels, apps, remote, volume, etc.).
  2. Several specialized widgets depending on the type of player (audio, video, casting…).

This is HA’s choice if I understand correctly

  1. An adaptive version, lightweight on mobile and more complete on desktop.

Personally, I like the first one…

Quick question, I don’t have a Sonos at home but did I understand correctly:

  • The existing music widget is only for Sonos?
  • Gladys already sees existing Sonos groups, but doesn’t allow creating or modifying them???

Base: 4 Matter clusters (OnOff, MediaPlayback, LevelControl, KeypadInput)

List of features for a « universal » media widget identified so far:

:bullseye: CORE FUNCTIONS (All Devices)

Playback Control

✅ play() - Start playback
✅ pause() - Pause
✅ stop() - Stop
✅ next() - Next song/program
✅ previous() - Previous song/program

Timeline / Progress

✅ seek(position) - Jump to a position
✅ displayTimeline() - Display progress bar
✅ displayTimeElapsed() - Display elapsed time (15:45)
✅ displayTimeRemaining() - Display remaining time (-14:15)

Volume Control

✅ setVolume(0-100) - Set volume
✅ increaseVolume() - Increase volume
✅ decreaseVolume() - Decrease volume
✅ mute() - Mute
✅ unmute() - Unmute
✅ displayVolumeLevel() - Display volume %

Power Management

✅ powerOn() - Turn on the device
✅ powerOff() - Turn off the device
✅ togglePower() - Toggle on/off
✅ displayPowerStatus() - Display power status

Device Info

✅ displayDeviceName() - Display device name
✅ displayStatus() - Display status (Playing, Paused, Stopped)
✅ displayCurrentSource() - Display current source

:musical_note: AUDIO PLAYER FUNCTIONS (Yoto, Sonos, etc.)

Metadata Display

✅ displayArtwork() - Display album cover
✅ displayArtist() - Display artist name
✅ displayTitle() - Display song title
✅ displayAlbum() - Display album name
✅ displayDuration() - Display total duration

Optional Audio Features

✅ shuffle() - Shuffle mode
✅ repeat() - Repeat mode
✅ like() - Like the song (♥)
✅ dislike() - Dislike
✅ showQueue() - Display playback queue

:television: VIDEO PLAYER FUNCTIONS (Freebox Player, TV, etc.)

Video Display

✅ displayVideoOutput() - Display video
✅ displayThumbnail() - Display thumbnail
✅ displayChannelLogo() - Display channel logo
✅ displayProgramTitle() - Display program title

Video-Specific Controls

✅ fastForward() - Fast forward
✅ rewind() - Rewind
✅ frameByFrame() - Frame-by-frame advance
✅ subtitles() - Enable/disable subtitles
✅ audioTracks() - Change audio track

:television: TV CHANNELS FUNCTIONS

Channel Selection

✅ setChannel(channelId) - Change channel (TMC, TF1, etc.)
✅ displayCurrentChannel() - Display current channel
✅ displayChannelNumber() - Display number (34)
✅ displayChannelName() - Display name (TMC)
✅ displayChannelLogo() - Display logo
✅ nextChannel() - Next channel (+)
✅ previousChannel() - Previous channel (-)

Channel Quick Access

✅ renderQuickAccessButtons() - [TF1] [france2] [TMC] [france3]
✅ setFavoriteChannel(channelId) - Mark as favorite
✅ removeFavoriteChannel(channelId) - Remove from favorites
✅ showAllChannels() - Modal with all channels (62+)
✅ searchChannel(name) - Search for a channel

EPG (Electronic Program Guide)

✅ displayCurrentProgram() - Display current program
✅ displayProgramTitle() - Program title (NCIS Los Angeles)
✅ displayProgramTime() - Time (15:45 - 16:30)
✅ displayProgramDescription() - Program synopsis
✅ displayNextPrograms() - Display upcoming programs
✅ displayProgramDuration() - Program duration

:clapper_board: STREAMING SERVICES / APPS FUNCTIONS

Application Launcher

✅ launchApp(appId) - Launch an app (Netflix, Disney+, etc.)
✅ displayAvailableApps() - Display list of available apps
✅ displayCurrentApp() - Display current app
✅ closeApp() - Close the app
✅ switchApp(appId) - Switch to another app

Streaming Services

✅ launchNetflix()
✅ launchDisneyPlus()
✅ launchAmazonPrime()
✅ launchAppleTV()
✅ launchYouTube()
✅ launchSpotify()
✅ launchAllApps() - View all apps

Test
Test1024×1536 158 KB

16

Hi @b3n.0!

Thanks for this exploration work, it’s exactly what I needed to decide. Here’s what I propose.

On the 3 approaches: we’re going with option 1

A single widget that adapts to the device’s capabilities. Quick note: this is actually what Home Assistant does. It doesn’t have multiple specialized widgets, it has a single media_player entity and a single card that displays/hides controls based on the device’s supported_features. So the market leader agrees with you :slightly_smiling_face:

Approach 3 (mobile/desktop) isn’t an alternative: it’s responsive, and it comes naturally with the dashboard grid.

The spec I propose

By digging into the existing code, I realized we already have two overlapping media models: the music category (used by Sonos, with playback state) and the television category (30 types, used only by remotes created manually via Broadlink/MQTT). Instead of creating a third one or merging everything, I propose giving each a clear role:

  • music = the media player (everything with a state: playback, volume). This is where Matter connects. Immediate benefit: your Freebox and Yoto will appear in the same widget as Sonos, which will be renamed “Media Player” and made adaptive.
  • television = the remote control (stateless buttons: arrows, OK, back). This is where Matter KeypadInput connects. Nice bonus: the “remote control” section of the widget will now also work for existing Broadlink infrared remotes, not just for Matter.
  • Power: OnOff cluster to switch.

And we’ll take this opportunity to modernize the model. Today, playback is represented by several separate features (play, pause, playback_state…). Since then, Gladys has gained supported_options: the ability for a feature to announce the list of values it supports, device by device (already used for air conditioning modes). We’ll use that here:

  • A single controllable playback_state feature: its value is the current state (playing / paused / stopped), synchronized in real time, and modifying it sends the corresponding command. Its supported_options list the states that your device can reach, automatically built from what the device declares (on the Matter side, each device publishes the exact list of commands it accepts via the AcceptedCommandList attribute).
  • The widget therefore displays exactly the play/pause/stop buttons that the device supports, with the active state highlighted, without hardcoding anything by brand or protocol. It’s the equivalent of Home Assistant’s supported_features, but in data.
  • next and previous remain separate features: they’re stateless impulses, not states.
  • In scenes, “set player to play” will become a real state assignment, more natural than a “button press”.

The complete Matter mapping:

Matter Cluster Gladys Feature
MediaPlayback Controllable music/playback_state, options from AcceptedCommandList, real-time state via the CurrentState attribute
MediaPlayback (Next/Previous if supported) music/next, music/previous
LevelControl (media device) music/volume normalized 0 to 100
KeypadInput television/up, down, left, right, enter, return
OnOff switch/binary

Two points I particularly care about:

  1. State feedback. The playback state will be read at startup and followed by subscription. Without that, the widget “guesses” the state and gets out of sync as soon as you use the real remote. It’s the kind of detail that makes the difference between an integration that works in demo and an integration that’s pleasant to use daily.
  2. Volume normalization to 0 to 100. Matter speaks in 0 to 254, Sonos in 0 to 100. By normalizing at the service boundary, “volume at 30” means the same thing in a scene regardless of the protocol. That’s what makes Gladys protocol-agnostic.

For existing Sonos users, a migration will automatically convert the old play/pause features to this new model, nothing to do on their side.

On your big feature list

It needs to be deflated, for a factual reason: Matter MediaPlayback exposes no metadata. No title, no artist, no artwork. And EPG doesn’t exist at all in Matter. The big widget with artwork can therefore not be fed by Matter today. Channels and application launching do have clusters (Channel, ApplicationLauncher) but almost no real device implements them. We’ll only touch that if your Freebox plugin exposes them and we can test in real life.

So phasing:

  1. Server: the new playback_state model + Sonos migration + the Matter mapping above
  2. Frontend: evolution of the Music widget into an adaptive “Media Player” (each section only appears if the features exist), and support for supported_options features in scenes
  3. Later, if real devices justify it: mute, position/seek, channels, applications, metadata on the Sonos side

Your Sonos questions

  • Yes, the music widget is de facto Sonos-only: Google Cast and AirPlay only expose notification playback, so they don’t pass the widget filter.
  • Sonos groups are not managed anywhere today, neither in read nor in write. It’s a subject in its own right, we’ll keep it for later.

Next steps

For implementation, Claude Fable 5 will work on the subject anyway, code is no longer the bottleneck. What really matters now is to validate the spec above, because that’s what will determine if the result is good.

And that’s where your input is most valuable:

  1. Validate (or challenge) this plan: does it cover your Freebox and Yoto use cases well?
  2. Real-world testing when it’s ready: you’re the only one with these devices on hand.

Let me know what you think!