Zigbee2mqtt: Docker test image based on Gladys v4

First, we need to validate that on GitHub, when we are contributors, there is no way to push PRs to the master branch. (I think that’s what we said yesterday).

I don’t think we will create a branch but rather everyone can help on a branch, but it’s not possible to push to master :slight_smile:

Exactly, the infamous « shitty repo »

1 Like

I think that’s what VonOx meant.

On my side, here’s what I imagined:

  • Someone devs a service on their side, for example Zigbee2Mqtt
  • Then creation of a « Zigbee2Mqtt » branch with an advanced dev version (functional but to be completed, tested and validated) on Gladys official
  • Everyone can propose PRs on this branch
  • Only those with the rights can approve a PR then merge to Master

For me, this clearly makes sense and will greatly facilitate contributions to a service under development.

Yes, that’s what we said yesterday during the developer call, but we need to test this on a repo to ensure that no one can merge into master (as we don’t want that).

I certainly hope so, otherwise I can’t even imagine the chaos :laughing:

I have a bit of time to help with this feature and to finalize the great work of @Reno
I will start by rebasing, merging your 2 PRs @lmilcent and pushing a PR on Gladys + new image.
Then, I need to list the last things to do (tests, bugs, …) and get to it :slight_smile:
In the meantime, it would be good to test the functionality you describe on a repo yes.

3 Likes

If you ever need help with something, don’t hesitate (test, bugsn …) :slight_smile:

1 Like

Same, I have Zigbee devices and I know NodeJS (+ Docker), I can help you validate part of it in my environment for example.

Thanks and good luck anyway :smiley:

1 Like

To you 4, the Zigbee2mqtt integration will be finished tonight!!

2 Likes

Guys, I don’t understand any of this technical language… but it’s moving, it’s awesome!!

Hello everyone and happy new year!

I see that I’m finally reconnecting at the right time…

Sorry to everyone, but I had to disconnect from the project very abruptly and haven’t had any time to get back to it for about 2 months…

I’m fully on board to work with others on this topic and had actually requested this several times in the past, given my limited available time.

However, I’m happy to continue giving my opinion on the changes because the choices I made, especially due to the use of Docker, may be difficult to understand.
Let me explain:
Thanks @lmilcent for your PRs but I’m not sure that these resolve the issue for everyone.
In the Zigbee2mqtt container, it is quite normal for the device to always be the same inside the container. It’s just a link to the device on the host machine that is passed to the container at launch and this simplifies things because the internal configuration of the container is always the same for everyone.
In fact, it’s the modification of the PathOnHost field in the file /src/server/services/zigbee2mqtt/docker/zigbee2mqtt-container.json, which passes the device to the container but it is always called /dev/ttyACM0 in the container. (I have a dongle on /dev/ttyUSB0 at home and it works well)
This is done here in the code:

So, it seems to me that your problem is a specific case that should be studied separately.

Give me your opinion on the subject.

Also, I had an issue with my PR on the Gladys repo and I had to recreate it when I « disappeared »…
All the Zigbee2Mqtt modifications could be grouped into 1 commit but separate commits would be needed for some modifications that I made to the core, in order to facilitate validation by @pierre-gilles.
From memory, this was about:

1 Like

Welcome back @Reno

I didn’t understand that at all, no matter the source port, you bind to /tty/ACM0 in the z2m container.

The problem with this is that to change it, you have to recreate the container.

Did I understand correctly?

Welcome back among us :wink:

I suspected you did that, keeping the same device path in the container but the binder to the real path on the OS.
But this case may cause a compatibility issue if a user eventually wants to clone the Gladys repo and run it without using Docker.

In any case, on my side, the container was launched with ttyACM0 on the OS side, which obviously caused an error when launching the container since the equipment did not exist.

I ask you the opposite question: why not keep the same access path as the « real » one?
In itself, if all the configuration is dynamic (as I propose with my PR), it shouldn’t be a problem.

Yes, you understood correctly.
In case the device is modified in the container’s configuration file, it will also need to be restarted to re-read it, I believe.

I made this choice:

  1. To avoid having different configurations for this Z2M container, which would be complex to debug remotely,
  2. To avoid sharing all devices with the Z2M container (which we do not control), but only the Z2M dongle.

Okay, that works for me, that makes sense :+1:

This, therefore, is a bug in the module because the Z2M container should not start if no device is selected, or else, the device name was not written correctly in the DB.

This required fewer file modifications (only the container configuration file and not the Z2M one). After that, it should also work, but I don’t understand something: if you have /dev/ttyACM0 in your dongle variable on the OS side, when you write it in the Z2M container, it’s still not correct, right?

During my last test, I had:

  1. Created a new Gladys DB
  2. Selected ttyUSB0
  3. Launched the creation of containers via the web interface

And the z2m container refused to start. With the modifications I proposed, it was perfect.

But I’m going to retest again, with a new container and a new Gladys DB to check. I’ll let you know in a few minutes.

Great. This will ensure it’s safe since no other user has had this issue.

What hardware/OS are you using?

Test effectué, le conteneur zigbee2mqtt ne se lance pas. z2m se lance correctement en revanche.
zigbee2mqtt ne génère même pas de logs.

J’ai lancé le nouveau conteneur sans base de données puisque c’était un test (donc pas de persistance) et avec puis sans la ligne -v /var/lib/gladysassistant:/var/lib/gladysassistant.

docker run -ti \
--log-opt max-size=10m \
--restart=always \
--privileged \
--network=host \
--name gladysDEBUG \
-e NODE_ENV=production \
-e SERVER_PORT=80 \
-e TZ=Europe/Paris \
-v /var/run/docker.sock:/var/run/docker.sock \
-v /dev:/dev \
-v /run/udev:/run/udev:ro \
r6n0/gladys-zigbee2mqtt

Voilà le détail des logs d’erreur :

Logs Gladys
2021-01-15T14:19:50+0100 <info> installMqttContainer.js:61 (Zigbee2mqttManager.installMqttContainer) Creating user/pass...
2021-01-15T14:19:50+0100 <info> installMqttContainer.js:73 (Zigbee2mqttManager.installMqttContainer) Zigbee2MQTT MQTT broker is starting...
2021-01-15T14:19:56+0100 <info> installMqttContainer.js:77 (Zigbee2mqttManager.installMqttContainer) MQTT broker container successfully started
2021-01-15T14:19:56+0100 <info> installZ2mContainer.js:25 (Zigbee2mqttManager.installZ2mContainer) Zigbee2mqtt is being installed as Docker container...
2021-01-15T14:19:56+0100 <info> installZ2mContainer.js:26 (Zigbee2mqttManager.installZ2mContainer) Pulling koenkk/zigbee2mqtt:latest image...
2021-01-15T14:19:57+0100 <info> installZ2mContainer.js:29 (Zigbee2mqttManager.installZ2mContainer) Preparing Zigbee2mqtt environment...
2021-01-15T14:19:57+0100 <info> installZ2mContainer.js:35 (Zigbee2mqttManager.installZ2mContainer) Creating container...
2021-01-15T14:19:57+0100 <info> installZ2mContainer.js:43 (Zigbee2mqttManager.installZ2mContainer) Zigbee2mqtt successfully installed as Docker container
2021-01-15T14:19:57+0100 <info> installZ2mContainer.js:55 (Zigbee2mqttManager.installZ2mContainer) Zigbee2mqtt is starting...
2021-01-15T14:19:58+0100 <error> installZ2mContainer.js:74 (Zigbee2mqttManager.installZ2mContainer) Zigbee2mqtt container failed to start: Error: (HTTP code 500) server error - Cannot restart container a1f72b47034f13c6169ea95e3cf961b08ae2d9ddcf50fca200888848b06d98ef: error gathering device information while adding custom device "/dev/ttyACM0": no such file or directory
    at /src/server/node_modules/docker-modem/lib/modem.js:257:17
    at getCause (/src/server/node_modules/docker-modem/lib/modem.js:287:7)
    at Modem.buildPayload (/src/server/node_modules/docker-modem/lib/modem.js:256:5)
    at IncomingMessage.<anonymous> (/src/server/node_modules/docker-modem/lib/modem.js:232:14)
    at IncomingMessage.emit (events.js:326:22)
    at endReadableNT (_stream_readable.js:1241:12)
    at processTicksAndRejections (internal/process/task_queues.js:84:21) {
  reason: 'server error',
  statusCode: 500,
  json: {
    message: 'Cannot restart container a1f72b47034f13c6169ea95e3cf961b08ae2d9ddcf50fca200888848b06d98ef: error gathering device information while adding custom device "/dev/ttyACM0": no such file or directory'
  }
}
2021-01-15T14:19:58+0100 <error> index.js:14 (process.<anonymous>) unhandledRejection catched: Promise {
  <rejected> Error: (HTTP code 500) server error - Cannot restart container a1f72b47034f13c6169ea95e3cf961b08ae2d9ddcf50fca200888848b06d98ef: error gathering device information while adding custom device "/dev/ttyACM0": no such file or directory
      at /src/server/node_modules/docker-modem/lib/modem.js:257:17
      at getCause (/src/server/node_modules/docker-modem/lib/modem.js:287:7)
      at Modem.buildPayload (/src/server/node_modules/docker-modem/lib/modem.js:256:5)
      at IncomingMessage.<anonymous> (/src/server/node_modules/docker-modem/lib/modem.js:232:14)
      at IncomingMessage.emit (events.js:326:22)
      at endReadableNT (_stream_readable.js:1241:12)
      at processTicksAndRejections (internal/process/task_queues.js:84:21) {
    reason: 'server error',
    statusCode: 500,
    json: {
      message: 'Cannot restart container a1f72b47034f13c6169ea95e3cf961b08ae2d9ddcf50fca200888848b06d98ef: error gathering device information while adding custom device "/dev/ttyACM0": no such file or directory'
    }
  }
}
2021-01-15T14:19:58+0100 <error> index.js:15 (process.<anonymous>) Error: (HTTP code 500) server error - Cannot restart container a1f72b47034f13c6169ea95e3cf961b08ae2d9ddcf50fca200888848b06d98ef: error gathering device information while adding custom device "/dev/ttyACM0": no such file or directory
    at /src/server/node_modules/docker-modem/lib/modem.js:257:17
    at getCause (/src/server/node_modules/docker-modem/lib/modem.js:287:7)
    at Modem.buildPayload (/src/server/node_modules/docker-modem/lib/modem.js:256:5)
    at IncomingMessage.<anonymous> (/src/server/node_modules/docker-modem/lib/modem.js:232:14)
    at IncomingMessage.emit (events.js:326:22)
    at endReadableNT (_stream_readable.js:1241:12)
    at processTicksAndRejections (internal/process/task_queues.js:84:21) {
  reason: 'server error',
  statusCode: 500,
  json: {
    message: 'Cannot restart container a1f72b47034f13c6169ea95e3cf961b08ae2d9ddcf50fca200888848b06d98ef: error gathering device information while adding custom device "/dev/ttyACM0": no such file or directory'
  }
}

Docker Inspect zigbee2mqtt :

Docker
pi@gladys:~ $ docker inspect zigbee2mqtt
[
    {
        "Id": "4fbcdacaf09340f9db83412b29ce4a26c1cc20a30e5dc978a8a6797297e11c85",
        "Created": "2021-01-15T13:29:57.209300743Z",
        "Path": "docker-entrypoint.sh",
        "Args": [
            "npm",
            "start"
        ],
        "State": {
            "Status": "created",
            "Running": false,
            "Paused": false,
            "Restarting": false,
            "OOMKilled": false,
            "Dead": false,
            "Pid": 0,
            "ExitCode": 128,
            "Error": "error gathering device information while adding custom device \"/dev/ttyACM0\": no such file or directory",
            "StartedAt": "0001-01-01T00:00:00Z",
            "FinishedAt": "0001-01-01T00:00:00Z"
        },
        "Image": "sha256:8a57907f6c4ecebea7358def33c3f54ac1d70c8fbbf6bc9e7bc8d1f940e9fbb4",
        "ResolvConfPath": "/var/lib/docker/containers/4fbcdacaf09340f9db83412b29ce4a26c1cc20a30e5dc978a8a6797297e11c85/resolv.conf",
        "HostnamePath": "/var/lib/docker/containers/4fbcdacaf09340f9db83412b29ce4a26c1cc20a30e5dc978a8a6797297e11c85/hostname",
        "HostsPath": "/var/lib/docker/containers/4fbcdacaf09340f9db83412b29ce4a26c1cc20a30e5dc978a8a6797297e11c85/hosts",
        "LogPath": "/var/lib/docker/containers/4fbcdacaf09340f9db83412b29ce4a26c1cc20a30e5dc978a8a6797297e11c85/4fbcdacaf09340f9db83412b29ce4a26c1cc20a30e5dc978a8a6797297e11c85-json.log",
        "Name": "/zigbee2mqtt",
        "RestartCount": 0,
        "Driver": "overlay2",
        "Platform": "linux",
        "MountLabel": "",
        "ProcessLabel": "",
        "AppArmorProfile": "",
        "ExecIDs": null,
        "HostConfig": {
            "Binds": [
                "/var/lib/gladysassistant/zigbee2mqtt/z2m:/app/data"
            ],
            "ContainerIDFile": "",
            "LogConfig": {
                "Type": "json-file",
                "Config": {}
            },
            "NetworkMode": "host",
            "PortBindings": {},
            "RestartPolicy": {
                "Name": "always",
                "MaximumRetryCount": 0
            },
            "AutoRemove": false,
            "VolumeDriver": "",
            "VolumesFrom": null,
            "CapAdd": null,
            "CapDrop": null,
            "CgroupnsMode": "host",
            "Dns": [],
            "DnsOptions": [],
            "DnsSearch": [],
            "ExtraHosts": null,
            "GroupAdd": null,
            "IpcMode": "private",
            "Cgroup": "",
            "Links": null,
            "OomScoreAdj": 0,
            "PidMode": "",
            "Privileged": false,
            "PublishAllPorts": false,
            "ReadonlyRootfs": false,
            "SecurityOpt": null,
            "UTSMode": "",
            "UsernsMode": "",
            "ShmSize": 67108864,
            "Runtime": "runc",
            "ConsoleSize": [
                0,
                0
            ],
            "Isolation": "",
            "CpuShares": 0,
            "Memory": 0,
            "NanoCpus": 0,
            "CgroupParent": "",
            "BlkioWeight": 0,
            "BlkioWeightDevice": [],
            "BlkioDeviceReadBps": null,
            "BlkioDeviceWriteBps": null,
            "BlkioDeviceReadIOps": null,
            "BlkioDeviceWriteIOps": null,
            "CpuPeriod": 0,
            "CpuQuota": 0,
            "CpuRealtimePeriod": 0,
            "CpuRealtimeRuntime": 0,
            "CpusetCpus": "",
            "CpusetMems": "",
            "Devices": [
                {
                    "PathOnHost": "/dev/ttyACM0",
                    "PathInContainer": "/dev/ttyACM0",
                    "CgroupPermissions": "rwm"
                }
            ],
            "DeviceCgroupRules": null,
            "DeviceRequests": null,
            "KernelMemory": 0,
            "KernelMemoryTCP": 0,
            "MemoryReservation": 0,
            "MemorySwap": 0,
            "MemorySwappiness": null,
            "OomKillDisable": false,
            "PidsLimit": null,
            "Ulimits": null,
            "CpuCount": 0,
            "CpuPercent": 0,
            "IOMaximumIOps": 0,
            "IOMaximumBandwidth": 0,
            "MaskedPaths": [
                "/proc/asound",
                "/proc/acpi",
                "/proc/kcore",
                "/proc/keys",
                "/proc/latency_stats",
                "/proc/timer_list",
                "/proc/timer_stats",
                "/proc/sched_debug",
                "/proc/scsi",
                "/sys/firmware"
            ],
            "ReadonlyPaths": [
                "/proc/bus",
                "/proc/fs",
                "/proc/irq",
                "/proc/sys",
                "/proc/sysrq-trigger"
            ]
        },
        "GraphDriver": {
            "Data": {
                "LowerDir": "/var/lib/docker/overlay2/d99dc73114e87665c992b2a548d43c1700aa7df2ca88643d06f9fb7f7c41a869-init/diff:/var/lib/docker/overlay2/80aa1b93c3f3ce9e58aff7a0e71b216ffa835a47b4b2c5097a625b8147d2b453/diff:/var/lib/docker/overlay2/838ab8fa2e9301bff69513f43fa4728623d955117284379be6dfacc6a5cff594/diff:/var/lib/docker/overlay2/54ca51484d1fa5efa7000e3f8e9ca86e7e0c8f58e507bf899de8d5a1a6890260/diff:/var/lib/docker/overlay2/337c5ff44785f260c0a7d62b79d8deab1f6b6236f47c197cdc6e629d48c02243/diff:/var/lib/docker/overlay2/2c5d4002df2d44ad8508f11431cef3ceb3704c30d22ea2439648cc4bb8f92e55/diff:/var/lib/docker/overlay2/8bbe4879392cd3ac7257f91e65399751571acf8f84230b637293bcd97ee763ac/diff:/var/lib/docker/overlay2/3557a8d8a1f38bc8dcaea362f62435c1dddf1e7f09ab1b681904bd664aebfc97/diff:/var/lib/docker/overlay2/047849d144fa5e49452ffe93f1e12d881524fdfca746b63e84eebc3d92ae9d93/diff:/var/lib/docker/overlay2/a4aae5f649b7020069af11ed9528930a917d13172ebb205a3257e56a664825a3/diff:/var/lib/docker/overlay2/0c5824017e1b39a0c82bdfacadc6c2670ca05f9ce7c372310adb4961bd9c4acb/diff:/var/lib/docker/overlay2/57ad8fb1dea139313619455a2bfdbab7875d480c4de01797326520df8bd6bdc4/diff:/var/lib/docker/overlay2/f4456017e082fa551e765733362324693d36b92cd0111619807c60b439a0fbd7/diff:/var/lib/docker/overlay2/876f3fe7a27660468d0d2a003f549c2f39ee660e138cdb91e1902f1cf5728309/diff:/var/lib/docker/overlay2/20c4c4b21539a77f1f066f4955a8b4f20bd687d4b80eb483672e68e4bab7e9e9/diff",
                "MergedDir": "/var/lib/docker/overlay2/d99dc73114e87665c992b2a548d43c1700aa7df2ca88643d06f9fb7f7c41a869/merged",
                "UpperDir": "/var/lib/docker/overlay2/d99dc73114e87665c992b2a548d43c1700aa7df2ca88643d06f9fb7f7c41a869/diff",
                "WorkDir": "/var/lib/docker/overlay2/d99dc73114e87665c992b2a548d43c1700aa7df2ca88643d06f9fb7f7c41a869/work"
            },
            "Name": "overlay2"
        },
        "Mounts": [
            {
                "Type": "bind",
                "Source": "/var/lib/gladysassistant/zigbee2mqtt/z2m",
                "Destination": "/app/data",
                "Mode": "",
                "RW": true,
                "Propagation": "rprivate"
            }
        ],
        "Config": {
            "Hostname": "gladys",
            "Domainname": "",
            "User": "",
            "AttachStdin": false,
            "AttachStdout": false,
            "AttachStderr": false,
            "Tty": false,
            "OpenStdin": false,
            "StdinOnce": false,
            "Env": [
                "PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin",
                "NODE_VERSION=12.20.0",
                "YARN_VERSION=1.22.5"
            ],
            "Cmd": [
                "npm",
                "start"
            ],
            "Image": "koenkk/zigbee2mqtt:latest",
            "Volumes": null,
            "WorkingDir": "/app",
            "Entrypoint": [
                "docker-entrypoint.sh"
            ],
            "OnBuild": null,
            "Labels": {}
        },
        "NetworkSettings": {
            "Bridge": "",
            "SandboxID": "f45106576f3b03452b3347db23ae157b91ea7d6a9e7dc335ffad1f5ac560df7d",
            "HairpinMode": false,
            "LinkLocalIPv6Address": "",
            "LinkLocalIPv6PrefixLen": 0,
            "Ports": {},
            "SandboxKey": "/var/run/docker/netns/default",
            "SecondaryIPAddresses": null,
            "SecondaryIPv6Addresses": null,
            "EndpointID": "",
            "Gateway": "",
            "GlobalIPv6Address": "",
            "GlobalIPv6PrefixLen": 0,
            "IPAddress": "",
            "IPPrefixLen": 0,
            "IPv6Gateway": "",
            "MacAddress": "",
            "Networks": {
                "host": {
                    "IPAMConfig": null,
                    "Links": null,
                    "Aliases": null,
                    "NetworkID": "52cf186cb21192c3c9aa88cb4c94bb770b3deff48b02aa4392285f1cf4504c39",
                    "EndpointID": "",
                    "Gateway": "",
                    "IPAddress": "",
                    "IPPrefixLen": 0,
                    "IPv6Gateway": "",
                    "GlobalIPv6Address": "",
                    "GlobalIPv6PrefixLen": 0,
                    "MacAddress": "",
                    "DriverOpts": null
                }
            }
        }
    }
]

Here is the error when launching zigbee2mqtt, visible in the container inspection.