Hello everyone,
Following the discussion https://community.gladysassistant.com/t/rebond-sur-le-sujet-transfert-zigbee2mqtt-ha-vers-gladys, I’m starting a Home Assistant to Gladys migration tool. This thread is to present the project, explain and discuss the few core modifications it requires, and gather your opinions before writing the first line of code.
Why
Leaving Home Assistant for Gladys today means starting everything over manually: devices, rooms, users, scenes… and losing your history. This exit cost deters many curious users. The goal is for a “standard” HA user to be able to migrate in less than a quarter of an hour, and to know precisely what cannot be migrated, and why.
What the tool will do
A service integrated into Gladys, accessible:
- from the first launch, next to “Restore a Gladys Plus backup”;
- then in Settings → Migration.
Version 1
- Analysis of the HA installation, with a device-by-device compatibility report: migratable, partial, via an alternative, non-migratable, ignored.
- Installation of necessary integrations from the store, by checking boxes.
- Devices (with their name and room), rooms, users, and service integrations (weather, calendars, notifications…).
- HA inputs (vacation mode, adjustable settings, counters) recreated as virtual devices.
- Presence: network and Bluetooth trackers linked to users, zones, arrivals, and departures.
- HA add-ons (Mosquitto, Zigbee2MQTT, Node-RED, Frigate…) associated with their Gladys equivalent: native service or store integration.
- Complete histories: long-term statistics and recent states, including energy counters, with an estimate of the required disk space.
- HA scenes (the “ambiences”: lamp at 20%, shutters closed…) and simple scripts, converted into Gladys scenes to be launched manually.
- Simple automations: schedules, sunrise and sunset, detected movement or opening, temperature thresholds… The others are presented in a readable form, with a pre-filled scene draft, so the user can recreate them themselves. It’s also a good way to discover Gladys’s scene editor.
- What doesn’t have an equivalent in the core (the sun, workdays…) can rely on a device you already have, on a virtual device, or on an integration: the tool simply asks which one to use.
- Dashboards: a “Home” dashboard is generated from what has been migrated (rooms, ambiences, weather, cameras, energy), so you never arrive on an empty page. Your HA dashboards are taken up again afterwards, in an approximate way.
Version 2
- A migration assistant for more advanced automations and parameterized scripts, which will evolve with Gladys’s scene engine.
A few principles
- Read-only on the HA side: the tool modifies nothing there.
- Two modes: coexistence (we try Gladys without stopping HA) or switch. We switch from one to the other by restarting the tool.
- Guaranteed cancellation: the tool only undoes what it has done, and puts Gladys back exactly in its previous state.
- Nothing leaves the instance. The HA token and passwords remain in memory, for the necessary time.
- Relaunchable: we relaunch the tool to retrieve what has changed in HA, without creating duplicates.
For devices without a Gladys integration, the tool first encourages the user to create it themselves, thanks to the official integration model. Otherwise, it offers to publish a request in “Feature Requests”, with a message already written.
How it works, briefly
The tool does not “create” devices. It installs the appropriate Gladys integration, lets it discover the devices, then matches each HA device with the discovered device using a hardware key: MAC address, Zigbee address (IEEE), serial number… It then applies the name and room from HA. This mode of operation works just as well with native integrations as with those from the store.
The study of the code of the 12 most used HA integrations shows that, for 8 of them, this matching is safe: Zigbee (Zigbee2MQTT and ZHA), Shelly, Tuya, Google Cast, UPnP, DLNA… None of them are entirely manual.
What I ask of the core (for discussion)
I tried to affect the core as little as possible. The necessary libraries (archive reading, YAML, decrypting recent HA backups) remain service-specific, like yaml for zigbee2mqtt.
| # | Modification | Why |
|---|---|---|
| 1 | Three new tables t_migration_run, t_migration_item and t_migration_history_cursor |
Keep track of what has been imported: restart without duplicates, resume after interruption, exact cancellation. No existing model is modified. The tables have a source column to one day serve other systems (Jeedom, Domoticz…). |
| 2 | A Settings → Migration page and a line on the home screen | Tool entry points |
| 3 | Reserved params GLADYS_HW_* (GLADYS_HW_MAC, GLADYS_HW_IEEE, GLADYS_HW_SERIAL, GLADYS_HW_UDN…), following the model of GLADYS_TRANSPORT |
Have each integration publish the hardware identity of its devices in a standard form. Matching then becomes generic, without rules to write per integration. Benefit beyond migration: identify the same device seen by two integrations, and propose device.migrate. GLADYS_*: it’s the SDK that will only send them to versions that accept them. |
| 4 | (Optional) A backup upload route for HA | Only useful if HA is down. In the common case, Gladys downloads the backup from HA (like for Gladys Plus), already decrypted by HA. The body of JSON requests is currently limited to 100 ko. |
Another point of attention: the daily purge of states (90 days by default) would erase the imported history. The tool will therefore offer to set the retention to “unlimited”, a setting that the user can modify later. This requires no core modification, but I prefer to mention it.
Core Evolution Paths (for V2 and beyond)
While studying the scene engine for HA automation conversion, I identified gaps. Filling them would directly increase what the tool can convert. They are not necessary for V1: these are proposals to discuss.
| # | Path | Interest |
|---|---|---|
| P1 | Scene execution modes (“ignore if already running”, “restart”, “queue”) | Today, all executions of a scene run in parallel. A scene with a wait can therefore launch several times at the same time. This is the biggest gain. |
| P2 | Action “stop a scene” | Timer-based triggers |
| P3 | Solar condition (day or night, before or after sunset) | Lighting, shutters. In practice, a “day/night” device already covers the need. A small “Sun” integration (day/night state, elevation, azimuth) would be welcome in the store. |
| P4 | Duration condition (“in this state for N minutes”) | Alerts |
| P5 | Original trigger of the scene usable in conditions | Scenes with multiple triggers and multiple behaviors |
| P6 | “Device reachable / recent value” condition or trigger (Gladys already calculates “no recent value”, but only for display) | “This sensor is no longer responding” alerts, behavior similar to HA when a device is unavailable |
By the way, I found a bug: a “MQTT message received” trigger whose topic contains a wildcard (+ or #) never triggers. Ticket: Scene trigger "MQTT message received" never fires when the topic contains a wildcard (+ or #) · Issue #3400 · GladysAssistant/Gladys · GitHub. A fix is proposed and awaits a maintainer’s review: fix: fire "MQTT message received" scene trigger on wildcard topics (#3400) by Pierre-Gilles · Pull Request #3401 · GladysAssistant/Gladys · GitHub.
What I will ask integration developers
Not much, and only in the next SDK update:
- publish the hardware identity of your devices with the help of the SDK (point 3 above);
- if needed, add to the root of your repository a small file
gladys-migration.json. It describes the exceptions (functions whose names differ from HA, unit conversions) and the settings to take from HA (address of a broker, ESPHome encryption key…). Theintegration-template-jsmodel will contain an example.
With @Will_71 and his tracker
He proposed a so-called mapping catalogue for HA → Gladys, which will rely on the integration tracker by @Will_71. It already links HA integrations with Gladys, entirely automatically. It would just need to add the collection of gladys-migration.json files and a « HA Migration » column. Once all integrations are created, the tracker would remain the tool that measures migration coverage. William, what do you think?
Where are we at
The V1 specifications are written, in French, awaiting their English version for the pull request to Gladys: GitHub - gboulvin/HA2Gladys-Migration-Tool · GitHub
| Section | Content |
|---|---|
| 00 — Vision | Objectives, scope, decisions |
| 01 — Architecture | Core service, tables, security, cancellation |
| 02 — Reading HA | API, backups (including encrypted), history |
| 03 — Correspondences | Catalogue, reconciliation, items, people, HA entries, add-ons |
| 04 — Scenes | HA scenes and scripts, simple automations, presence, substitutes |
| 05 — Paths | The assistant’s screens |
| 06 — Tests | 100% coverage, CI for monitoring HA versions |
| 07 — Histories | Import, energy, purge |
| 08 — Dashboards | Generated home, resuming HA dashboards |
All the work is intended to be submitted to the GladysAssistant organization once merged.
How you can help
- Your opinions on core changes, especially the
GLADYS_HW_*params. - Real HA backups, which I know how to anonymize with a provided script (names, addresses, serial numbers replaced, secrets removed). They are used to test on real installations and measure performance. They will never be published.
- Testers for the test Docker image, which will be announced here before the final pull request.
No development will be undertaken without validation of the principle. I therefore call on Gladys and/or HA users and developers to comment (here or on GitHub), propose…
Thanks in advance for your feedback!