Zigbee2mqtt: Docker test image based on Gladys v4

cicoub13 · March 10, 2021, 6:59pm

Good evening everyone. After long weeks of waiting , the work on Zigbee2Mqtt is finally stable (and 100% tested ).

You can therefore download the image and test it on your Raspberry Pi (it will be port 1080).
All feedback is welcome (especially on devices that do not work AND on the user experience).

A big thank you to @Reno for the base code and @lmilcent for the 150 test images

--log-opt max-size=10m \
--restart=always \
--privileged \
--network=host \
--name gladys-dev-zigbee2mqtt \
-e NODE_ENV=production \
-e SERVER_PORT=1080 \
-e TZ=Europe/Paris \
-e SQLITE_FILE_PATH=/var/lib/gladysassistant/gladys-zigbee2mqtt.db \
-v /var/run/docker.sock:/var/run/docker.sock \
-v /var/lib/gladysassistant:/var/lib/gladysassistant \
-v /dev:/dev \
-v /run/udev:/run/udev:ro \
cicoub13/gladys:dev-zigbee2mqtt

Sorry @VonOx but I will work on adding a pre-configured MQTT server later.

I clean up the code a bit and the PR should arrive by the end of the week.

VonOx · March 10, 2021, 7:02pm

No problem, good job

Albenss · March 10, 2021, 7:13pm

Hello, thank you for everyone’s work!
I’m trying to test this tonight and I’ll get back to you as soon as possible

Albenss · March 10, 2021, 8:42pm

Good evening, quick update, on the MQTT side I can see the connections being made.
On the Z2M side, I can see the actions of my peripherals (two switches WXKG01LM Aqara as well as a cube MFKZQ01LM Aqara).

However, on the Gladys side, the information does not appear on the dashboard. In the logs, it seems that the association of the physical device to the Gladys device is not working properly for the three devices:

2021-03-10T21:17:21+0100 <info> installZ2mContainer.js:25 (Zigbee2mqttManager.installZ2mContainer) Zigbee2mqtt is being installed as Docker container...
2021-03-10T21:17:21+0100 <info> installZ2mContainer.js:26 (Zigbee2mqttManager.installZ2mContainer) Pulling koenkk/zigbee2mqtt:latest image...
2021-03-10T21:17:50+0100 <info> installZ2mContainer.js:29 (Zigbee2mqttManager.installZ2mContainer) Preparing Zigbee2mqtt environment...
2021-03-10T21:17:50+0100 <info> installZ2mContainer.js:38 (Zigbee2mqttManager.installZ2mContainer) Configuration of Device /dev/ttyACM0
2021-03-10T21:17:50+0100 <info> installZ2mContainer.js:41 (Zigbee2mqttManager.installZ2mContainer) Creation of container...
2021-03-10T21:17:52+0100 <info> installZ2mContainer.js:44 (Zigbee2mqttManager.installZ2mContainer) Zigbee2mqtt successfully installed as Docker container
2021-03-10T21:17:52+0100 <info> installZ2mContainer.js:57 (Zigbee2mqttManager.installZ2mContainer) Zigbee2mqtt is starting...
2021-03-10T21:17:58+0100 <info> installZ2mContainer.js:69 (Zigbee2mqttManager.installZ2mContainer) Zigbee2mqtt successfully started
2021-03-10T21:17:58+0100 <info> connect.js:23 (MqttClient.<anonymous>) Connected to MQTT container mqtt://localhost:1884
2021-03-10T21:17:58+0100 <info> subscribe.js:12 (Zigbee2mqttManager.subscribe) Subscribing to MQTT topic zigbee2mqtt/#
2021-03-10T21:20:15+0100 <warn> handleMqttMessage.js:83 (Zigbee2mqttManager.handleMqttMessage) Zigbee2mqtt topic zigbee2mqtt/bridge/config/devices/get empty message.
2021-03-10T21:20:21+0100 <warn> handleMqttMessage.js:104 (Zigbee2mqttManager.handleMqttMessage) Zigbee2mqtt device 0x00158d00027bb168 not configured in Gladys.
2021-03-10T21:20:23+0100 <warn> handleMqttMessage.js:104 (Zigbee2mqttManager.handleMqttMessage) Zigbee2mqtt device 0x00158d00027bb168 not configured in Gladys.
2021-03-10T21:20:25+0100 <warn> handleMqttMessage.js:83 (Zigbee2mqttManager.handleMqttMessage) Zigbee2mqtt topic zigbee2mqtt/bridge/config/devices/get empty message.
2021-03-10T21:22:50+0100 <warn> handleMqttMessage.js:83 (Zigbee2mqttManager.handleMqttMessage) Zigbee2mqtt topic zigbee2mqtt/bridge/config/devices/get empty message.
2021-03-10T21:22:51+0100 <warn> handleMqttMessage.js:83 (Zigbee2mqttManager.handleMqttMessage) Zigbee2mqtt topic zigbee2mqtt/bridge/config/devices/get empty message.
2021-03-10T21:22:54+0100 <warn> handleMqttMessage.js:83 (Zigbee2mqttManager.handleMqttMessage) Zigbee2mqtt topic zigbee2mqtt/bridge/config/devices/get empty message.
2021-03-10T21:22:57+0100 <warn> handleMqttMessage.js:104 (Zigbee2mqttManager.handleMqttMessage) Zigbee2mqtt device 0x00158d0003748a21 not configured in Gladys.
2021-03-10T21:23:00+0100 <warn> handleMqttMessage.js:83 (Zigbee2mqttManager.handleMqttMessage) Zigbee2mqtt topic zigbee2mqtt/bridge/config/devices/get empty message.
2021-03-10T21:23:47+0100 <warn> handleMqttMessage.js:104 (Zigbee2mqttManager.handleMqttMessage) Zigbee2mqtt device 0x00158d0003748a46 not configured in Gladys.
2021-03-10T21:23:48+0100 <warn> handleMqttMessage.js:83 (Zigbee2mqttManager.handleMqttMessage) Zigbee2mqtt topic zigbee2mqtt/bridge/config/devices/get empty message.
2021-03-10T21:27:39+0100 <warn> handleMqttMessage.js:100 () Zigbee2mqtt device 0x00158d00027bb168, feature action not configured in Gladys.
2021-03-10T21:27:39+0100 <warn> handleMqttMessage.js:100 () Zigbee2mqtt device 0x00158d00027bb168, feature battery not configured in Gladys.
2021-03-10T21:27:39+0100 <warn> handleMqttMessage.js:100 () Zigbee2mqtt device 0x00158d00027bb168, feature linkquality not configured in Gladys.
2021-03-10T21:27:39+0100 <warn> handleMqttMessage.js:100 () Zigbee2mqtt device 0x00158d00027bb168, feature voltage not configured in Gladys.
2021-03-10T21:28:43+0100 <warn> handleMqttMessage.js:100 () Zigbee2mqtt device 0x00158d0003748a21, feature action not configured in Gladys.
2021-03-10T21:28:43+0100 <warn> handleMqttMessage.js:100 () Zigbee2mqtt device 0x00158d0003748a21, feature battery not configured in Gladys.
2021-03-10T21:28:43+0100 <warn> handleMqttMessage.js:100 () Zigbee2mqtt device 0x00158d0003748a21, feature linkquality not configured in Gladys.
2021-03-10T21:28:43+0100 <warn> handleMqttMessage.js:100 () Zigbee2mqtt device 0x00158d0003748a21, feature voltage not configured in Gladys.
2021-03-10T21:28:43+0100 <warn> handleMqttMessage.js:100 () Zigbee2mqtt device 0x00158d0003748a21, feature battery not configured in Gladys.
2021-03-10T21:28:43+0100 <warn> handleMqttMessage.js:100 () Zigbee2mqtt device 0x00158d0003748a21, feature click not configured in Gladys.
2021-03-10T21:28:43+0100 <warn> handleMqttMessage.js:100 () Zigbee2mqtt device 0x00158d0003748a21, feature linkquality not configured in Gladys.
2021-03-10T21:28:43+0100 <warn> handleMqttMessage.js:100 () Zigbee2mqtt device 0x00158d0003748a21, feature voltage not configured in Gladys.
2021-03-10T21:28:43+0100 <warn> handleMqttMessage.js:100 () Zigbee2mqtt device 0x00158d0003748a46, feature action not configured in Gladys.
2021-03-10T21:28:43+0100 <warn> handleMqttMessage.js:100 () Zigbee2mqtt device 0x00158d0003748a46, feature battery not configured in Gladys.
2021-03-10T21:28:43+0100 <warn> handleMqttMessage.js:100 () Zigbee2mqtt device 0x00158d0003748a46, feature linkquality not configured in Gladys.
2021-03-10T21:28:43+0100 <warn> handleMqttMessage.js:100 () Zigbee2mqtt device 0x00158d0003748a46, feature voltage not configured in Gladys.
2021-03-10T21:28:43+0100 <warn> handleMqttMessage.js:100 () Zigbee2mqtt device 0x00158d0003748a46, feature battery not configured in Gladys.
2021-03-10T21:28:43+0100 <warn> handleMqttMessage.js:100 () Zigbee2mqtt device 0x00158d0003748a46, feature click not configured in Gladys.

I specify that the three devices have only one functionality « Switch » feature.

pierre-gilles · March 11, 2021, 1:38am

Great job

Before submitting your PR, make sure to do a full code review by comparing with the Gladys master:

Are the modified files actually related to the zigbee2mqtt integration and not unrelated files?
Have any unnecessary dependencies been added?

Next, we will move on to the technical review of the PR (I would appreciate any help with this, everyone is welcome to review the code)

In parallel, any functional feedback is also welcome to ensure that the PR works as expected!

I hope it will be merged quickly into Gladys Well done for the work done!

cicoub13 · March 11, 2021, 9:06am

I updated the image with the correct features for these two devices.

Albenss · March 11, 2021, 9:37am

Indeed, the issue is resolved, everything works correctly for these two types of devices, the data is correctly displayed on the Dashboard.
Ultra-fast correction, well done

Just need to test the operation with scenes now

Albenss · March 15, 2021, 8:35pm

Good evening,
I have been trying for some time to launch Gladys Zigbee edition via docker-compose
rather than via simple docker.

There are however two problems with the MQTT container that is created and launched by the Z2M service.

The first problem seems to be a Linux permission issue, the mosquitto.conf file fails to create in the MQTT folder. The MQTT configuration file folder does not seem to be accessible by the container.

The second problem is related to the persistence file paths of Z2M and MQTT. When creating the image, it is possible to change the persistence path to avoid having it in « /var/lib/gladys… » the problem is that the mqtt and z2m folders do not follow the change and still register in the folder « /var/lib/gladysassistant/ ».

On the official Gladys image, the Mqtt container does not have these two problems, I have not yet looked at the difference between the two container management methods but it would be interesting to keep the same way of creating and launching the mosquitto container for Mqtt.

Here is the docker-compose in question if you have time to try to
reproduce the problem:

version: "2.4"

services:
  gladys-z2m:
    image: cicoub13/gladys:dev-zigbee2mqtt
    container_name: Gladys_Z2M_DEV
    restart: always
    privileged: true
    logging:
      options:
        max-size: "10m"
    network_mode: host
    environment:
      - "TZ=${TZ}"
      - "NODE_ENV=production"
      - "SERVER_PORT=1080"
      - "SQLITE_FILE_PATH=/var/lib/gladys-z2m/gladys-z2m-production.db"
    volumes:
      - "/var/run/docker.sock:/var/run/docker.sock"
      - "/dev:/dev"
      - "/run/udev:/run/udev:ro"
      - "./data/gladys_zigbee:/var/lib/gladys-z2m"
    labels:
      - "com.centurylinklabs.watchtower.enable=true"

Edit: I tried via the official Gladys image, as well as the HypriotOS and it’s the same problem.

pierre-gilles · March 16, 2021, 12:30pm

Indeed, I agree it’s a problem that these files are hardcoded, I had the same problem trying to test these integrations on an instance where Gladys already exists: it’s impossible

cicoub13 · March 16, 2021, 1:55pm

It’s exactly the same with the MQTT container proposed by Gladys (the code is inspired by it).
But I’ll check

pierre-gilles · March 17, 2021, 1:00am

I know it’s a problem on both sides

VonOx · March 17, 2021, 7:41am

An idea like that, maybe we should recover the variable
SQLITE_FILE_PATH and store the config in the same place, rather than hardcoding everything

pierre-gilles · March 22, 2021, 2:25am

Hello everyone!

For your information, following the changes made by @cicoub13 on the PR, I have conducted a technical and functional review of the PR.

My review report is available on GitHub: https://github.com/GladysAssistant/Gladys/pull/1098#issuecomment-803718817

Edit: I’m also waiting for the fix here:

cicoub13 · March 23, 2021, 10:59pm

As mentioned in the PR:

SQLITE_FILE_PATH management:white_check_mark:
Fix for button click values:white_check_mark:

I still have the small demo video and the documentation on gladys-website to do

Albenss · March 24, 2021, 3:00pm

Hello, great work, on my side just a remark, it seems that some mappings have been skipped for the two only types of devices I have (two switches WXKG01LM Aquara and a cube MFKZQ01LM Aquara).

The issue had been fixed but it seems that it is no longer the case.
Otherwise, I think the points mentioned above are well fixed, in any case, I haven’t seen any other issues

cicoub13 · March 24, 2021, 4:23pm

Hello @Albenss

I just fixed the type of your two devices. Can you update the image and test again?
Currently, the values single, double, hold are converted to numeric values.

The others are treated as strings: triple, quadruple, release, many and all those of your cube shake, wakeup, fall, tap, slide, flip180, flip90, rotate_left, rotate_right.

You can post/view the following thread: Scène avec un bouton (double ou simple clic)

Albenss · March 24, 2021, 5:00pm

So as a test, I started by deleting the Gladys container (but not Z2M and MQTT) to recreate it after pulling the new image. I then observed the same behavior and thought it was probably a problem with the database.

I therefore recreated everything with persistence in a new folder. Unfortunately, the automatic creation of the MQTT container does not work. It seems there is still an access rights issue with the folder, or MQTT does not know where to find its configuration file. In the MQTT log, I have this:
1616604311: Error: Unable to open config file /mosquitto/config/mosquitto.conf.

I think there are still places where the path is hardcoded. Indeed, when I change the persistence folder to /var/lib/z2m-test, when the zeegbee2mqtt service is activated, a new folder is created here /var/lib/gladysassistant.

I therefore have a duplicate folder created by the service:

$ ls -al gladysassistant/zigbee2mqtt/
mqtt/ z2m/

$ ls -al z2m-test/zigbee2mqtt/
mqtt/ z2m/

EDIT: However, it is indeed in the new persistence folder /var/lib/z2m-test/zigbee2mqtt/mqtt/ that the Mosquitto configuration files are located and not in the duplicate folder in the path /var/lib/gladysassistant. Therefore, the old path is probably hardcoded in MQTT.

cicoub13 · March 24, 2021, 5:26pm

Yes, sorry. You need to delete the devices so they have the correct feature types in the database.

For the paths and permissions, have you deleted all the containers since the evening of March 22? Otherwise, the Z2M and MQTT containers are still on the old paths.
The fix I made modifies the paths at creation, but does not handle post-creation changes.

Albenss · March 24, 2021, 5:28pm

Yes, I had indeed deleted all the containers. I will try again in 30 minutes after also deleting the images to test.

Albenss · March 24, 2021, 6:11pm

Still the same problem unfortunately…

Topic		Replies	Views
Gladys Assistant 4.2 est disponible avec la compatibilité Zigbee2mqtt ! 🚀 Actualités	146	5684	May 17, 2021
Image Node 18 : Tests intégration Bluetooth Développement	41	1260	December 13, 2022
Zigbee2mqtt - services démarrés mais croix rouge pour le lien Configuration	60	1831	May 13, 2023
Gladys et docker mqtt+zigbee2mqtt existant Configuration	14	414	November 13, 2023
Pouvoir utiliser un Zigbee2mqtt externe dans Gladys [Archive] Demande de fonctionnalités zigbee2mqtt , feature-integration	67	795	May 6, 2024

Zigbee2mqtt: Docker test image based on Gladys v4

Related topics