Zigbee2mqtt: Docker test image based on Gladys v4

You are using the SQLITE_FILE_PATH variable correctly (as in your docker-compose above)?

"SQLITE_FILE_PATH=/var/lib/gladys-z2m/gladys-z2m-production.db"

Can you give me the result of the following commands?

docker inspect -f '{{ .Mounts }}' gladys-z2m-mqtt

docker inspect -f '{{ .Mounts }}' gladys-z2m-zigbee2mqtt

My command is as follows:

    docker run -d \
    --log-opt max-size=10m \
    --restart=always \
    --privileged \
    --network=host \
    --name GladysZigbee2mqtt \
    -e NODE_ENV=production \
    -e SERVER_PORT=1080 \
    -e TZ=Europe/Paris \
    -e SQLITE_FILE_PATH=/var/lib/gladysassistant/gladys-zigbee2mqtt.db \
    -v /var/run/docker.sock:/var/run/docker.sock \
    -v /var/lib/z2m-test:/var/lib/gladysassistant \
    -v /dev:/dev \
    -v /run/udev:/run/udev:ro \
    cicoub13/gladys:dev-zigbee2mqtt
$ docker inspect -f '{{ .Mounts }}' gladys-z2m-mqtt
json
[{volume 227a9e1e87d118c13a7d214c28395292e7b69fae2abcf2b330821ceb31add32d /var/lib/docker/volumes/227a9e1e87d118c13a7d214c28395292e7b69fae2abcf2b330821ceb31add32d/_data /mosquitto/data local  true } {bind  /var/lib/gladysassistant/zigbee2mqtt/mqtt /mosquitto/config   true rprivate} {volume c30bc5eede272b937d1e6bb1dd46c224f576b9d3c294bec90ed8cd6cca80926c /var/lib/docker/volumes/c30bc5eede272b937d1e6bb1dd46c224f576b9d3c294bec90ed8cd6cca80926c/_data /mosquitto/log local  true }]

Regarding the second command, I cannot provide you with an answer since Z2M was not created.

$ docker inspect -f '{{ .Mounts }}' gladys-z2m-zigbee2mqtt
Error: No such object: gladys-z2m-zigbee2mqtt

OK, I understand :sweat_smile:

When you set these two options:

  • -v /var/lib/z2m-test:/var/lib/gladysassistant
  • SQLITE_FILE_PATH=/var/lib/gladysassistant/gladys-zigbee2mqtt.db

You mount your host folder /var/lib/z2m-test into the container as /var/lib/gladysassistant
Then you tell Gladys to work with /var/lib/gladysassistant, which it does when it creates the Mosquito and Z2M configuration files. That’s why you find the configuration files in /var/lib/z2m-test on your host.

When Gladys creates the other containers (Mosquito and Z2M), it uses the SQLITE_FILE_PATH variable to mount the volumes. So the /var/lib/gladysassistant of your host… That’s where it doesn’t work. I don’t know how to detect that /var/lib/gladysassistant is actually a mount of /var/lib/z2m-test.

I suggest you try with this docker-compose file :thinking:

docker run -d \
    --log-opt max-size=10m \
    --restart=always \
    --privileged \
    --network=host \
    --name GladysZigbee2mqtt \
    -e NODE_ENV=production \
    -e SERVER_PORT=1080 \
    -e TZ=Europe/Paris \
    -e SQLITE_FILE_PATH=/var/lib/z2m-test/gladys-zigbee2mqtt.db \
    -v /var/run/docker.sock:/var/run/docker.sock \
    -v /var/lib/z2m-test:/var/lib/z2m-test \
    -v /dev:/dev \
    -v /run/udev:/run/udev:ro \
    cicoub13/gladys:dev-zigbee2mqtt

The configuration files should be written by Gladys somewhere on your host. I decided to use SQLITE_FILE_PATH as the folder where to write these files. But without thinking about the case where you mount this folder from another path on your host.

But maybe we should use another variable for this? Like HOST_PATH_Z2M_FILES?

We arrive in the exotic, in standard it’s /var/lib/gladysassistant

We won’t handle all cases and we won’t be able to add a variable to production instances

I disagree, all the images I’ve used so far allow you to use a different persistence path than the official one.

The real issue we have here is having to specify multiple paths if we use an alternative path.

Do we really need to specify a path for the database, can’t we simply put all the files within the container in the same folder, here surely /var/lib/gladysassistant, thus set the value of the SQLITE_FILE_PATH (and therefore remove it from the config) and then simply ask the user to fill in the value PERSISTANCE_FOLDER (for example) which would map between the container folder and the host?

Here we are in a case where if we want to test several gladys images, we have to modify all the config paths, the SQLITE_FILE_PATH then the folder in the container and the mapping folder on the host.

Indeed, everything is working well now. It’s true that, as my message above indicates, we had to change three values to do this, but all the issues are resolved.

The two types of Aquara devices I have are working well, the file paths have been modified correctly, and the information is being uploaded to the dashboard as it should. Kudos to everyone who worked on this integration. :slight_smile:

Relis moi, j’ai pas dit que techniquement c’était pas possible. J’ai juste dit qu’on ne peut pas exploser toutes les instances de gladys en production. Car faire ce changement ça pète tout.

Exact specific case, not in production. The command from @cicoub13 works perfectly because all the paths match.

In short, it’s legacy.

Why not just do like the port exposure with Docker?
For example, by default an application starts on port 3456, but you want to expose port 80 publicly, then you use the option -p 80:3456.

For me, the same should be done in Gladys: everything is stored in the container by default in /var/lib/gladysassistant and the path modification goes through the modification of the container path: -v /data/exemple/Gladys:/var/lib/gladysassistant.

It seems simpler to do and understand!

Oh, that’s the case for Gladys, the problem is not there.

It would be necessary to be able to retrieve the bind (-v) from the container to communicate it to other containers. If we manage that, it will solve everything.

Goal: retrieve the bind from the container on /var/lib/gladysassistant

@cicoub13 I just quickly tested a small piece of js with the lib we use without going all the way.

var Docker = require('dockerode');
var docker = new Docker({socketPath: '/var/run/docker.sock'});
// HERE we can replace gladys with os.hostname
var container = docker.getContainer('gladys');

 container.inspect(function (err, data) {
         var string = JSON.stringify(data);
         var objectValue = JSON.parse(string);
         console.log(objectValue['HostConfig']['Binds']);
});

Returns the standard mounts

[
  '/var/run/docker.sock:/var/run/docker.sock',
  '/var/lib/gladysassistant:/var/lib/gladysassistant'
]

A small regex to get the host side bind and that should solve this point?

Not bad. I like the idea. But how do I get the right one? Since you can mount any folder from the host and any folder in Docker (with SQLITE_FILE_PATH).
And even mount multiple folders.

And it starts to get complicated :smile:

I feel like saying SQL is the DB, whatever.

Gladys, in any case (for the MQTT and Zigbee part), must write its config in /var/lib/gladysassystant, that’s in the container.
By recovering the bind on the host side, we can use it to create other containers and manage persistence.

My previous post is not a good example because I bind the same path. But for example, if I take the case of @Albenss

-v /var/lib/z2m-test:/var/lib/gladysassistant

My bit of JS will return

[
  '/var/run/docker.sock:/var/run/docker.sock',
  '/var/lib/z2m-test:/var/lib/gladysassistant'
]

By the way, I quickly tested a regex (which I couldn’t implement in JS but that’s not the point :sweat_smile: )

EDIT: regex101 offers a code generator to test. :expressionless: and it does the job

@VonOx that seems like a good solution to me :slight_smile: Indeed, this would avoid forcing the user to put this data in /var/lib/gladysassistant on the host if they don’t have that folder available.

The only remark is that on the JS side, you’ll need to run the regex with the basePath and not /var/lib/gladysassistant, and we’ll be good to go!

@cicoub13 are you integrating this into the Zigbee2mqtt PR?

Apart from that, I’m good to merge the zigbee2mqtt PR, I saw your video @cicoub13 and it’s amazing! It’s really great work, nothing to say :slight_smile: Congrats to everyone who worked on this PR, it’s amazing!

The basepath? Sorry, I didn’t understand

EDIT: Ah, the SQLITE_FILE_PATH?

I can also make a PR quickly to add the function in the lib/system to avoid loading cicoub’s one

EDIT2: Based on getNetworkMode it’s messy because eslint complains a bit but here’s a base @cicoub13

const get = require('get-value');
const { PlatformNotCompatible } = require('../../utils/coreErrors');
const { exec } = require('../../utils/childProcess');

/**
 * @description Get Gladys host bind folder from docker environment.
 * @returns {Promise} Resolve with host path.
 * @example
 * const getDockerHostFolder = await this.getDockerHostFolder();
 */
async function getDockerHostFolder() {
  if (!this.dockerode) {
    throw new PlatformNotCompatible('SYSTEM_NOT_RUNNING_DOCKER');
  }

  if (!this.dockerHostFolder) {
    const regex = /([\\/[\\w-]+]*):\\/var\\/lib\\/gladysassistant/gm;
    const cmdResult = await exec('head -1 /proc/self/cgroup | cut -d/ -f3');
    const [containerId] = cmdResult.split('\\n');
    const gladysContainer = this.dockerode.getContainer(containerId);
    const gladysContainerInspect = await gladysContainer.inspect();
    const bindPaths = get(gladysContainerInspect, 'HostConfig.Binds');
    const matches = bindPaths.matchAll(regex);
    this.dockerHostFolder = matches[0];
  }

  return this.dockerHostFolder;
}

module.exports = {
  getDockerHostFolder,
};

Yes, use the basePath function that @cicoub13 made in his PR which uses the SQLITE_FILE_PATH to find the folder used internally.

It’s in progress :wink: it’s complicated to test all cases (several dockers containing the word gladys, no mounted volume, …)

You need to do it by ID (like the getnetworkmode function)

That’s where the mistake is!

Internally, it should always be /var/lib/gladysassistant. (standard to be defined)
This variable (SQLITE_FILE_PATH) must be set by default in the image with this path.

Here we need to manage this variable (used only for the database) and the persistence bind.

For creating service containers, only the host bind matters, it doesn’t matter where the database is in the container.

There are 2 topics:

  • How to manage gladys « data » in the image → /var/lib/gladysassistant as standard
  • How to create service containers that use the same host bind path as gladys → the function I propose (/???/??? bound to /var/lib/gladysassistant)

No, because on our local machines, or outside Docker, you can put it wherever you want!

After that, it’s true that outside Docker or on our machines, we don’t pop containers.