Disk usage by Mosquitto logs

lmilcent · January 11, 2023, 10:34am

Hello everyone, especially @cicoub13 and @pierre-gilles,

I received an alert on my Gladys instance (mini-PC) because disk space was low.
Investigating, it’s caused by a container.

This container corresponds to eclipse-mosquitto, used by the Zigbee2MQTT service or by MQTT, I’m not certain.
This is confirmed by the info file at the source of this folder:

Digging a little further, restarting the container changed nothing. In fact, it’s a single file that’s using all the space:

/var/lib/docker/containers/9991e6bed47295b41c61b190cfb4168c0a2fe282b561e4afa950ceb8a2dbf781/9991e6bed47295b41c61b190cfb4168c0a2fe282b561e4afa950ceb8a2dbf781-json.log

This log file uses 24 GB!

Here’s an excerpt:

{\"log\":\"1668743198: Sending PUBLISH to mqttjs_85adae46 (d0, q0, r0, m0, 'zigbee2mqtt/capteurQualitéDeLair', ... (35 bytes))\\n\",\"stream\":\"stderr\",\"time\":\"2022-11-18T03:46:38.13707785Z\"}
{\"log\":\"1668743198: Received PUBLISH from mqttjs_85adae46 (d0, q0, r0, m0, 'zigbee2mqtt/bridge/logging', ... (283 bytes))\\n\",\"stream\":\"stderr\",\"time\":\"2022-11-18T03:46:38.650861947Z\"}
{\"log\":\"1668743198: Sending PUBLISH to gladys-main-instance-362751 (d0, q0, r0, m0, 'zigbee2mqtt/bridge/logging', ... (283 bytes))\\n\",\"stream\":\"stderr\",\"time\":\"2022-11-18T03:46:38.650892565Z\"}
{\"log\":\"1668743198: Sending PUBLISH to gladys-main-instance-372063 (d0, q0, r0, m0, 'zigbee2mqtt/bridge/logging', ... (283 bytes))\\n\",\"stream\":\"stderr\",\"time\":\"2022-11-18T03:46:38.650919128Z\"}
{\"log\":\"1668743198: Sending PUBLISH to NodeRed-6dda341ee7a029cb (d0, q0, r0, m0, 'zigbee2mqtt/bridge/logging', ... (283 bytes))\\n\",\"stream\":\"stderr\",\"time\":\"2022-11-18T03:46:38.650951475Z\"}
{\"log\":\"1668743198: Sending PUBLISH to mqttjs_85adae46 (d0, q0, r0, m0, 'zigbee2mqtt/bridge/logging', ... (283 bytes))\\n\",\"stream\":\"stderr\",\"time\":\"2022-11-18T03:46:38.651049516Z\"}
{\"log\":\"1668743198: Received PUBLISH from mqttjs_85adae46 (d0, q0, r0, m0, 'zigbee2mqtt/bridge/logging', ... (135 bytes))\\n\",\"stream\":\"stderr\",\"time\":\"2022-11-18T03:46:38.651070441Z\"}
{\"log\":\"1668743198: Sending PUBLISH to gladys-main-instance-362751 (d0, q0, r0, m0, 'zigbee2mqtt/bridge/logging', ... (135 bytes))\\n\",\"stream\":\"stderr\",\"time\":\"2022-11-18T03:46:38.651075885Z\"}
{\"log\":\"1668743198: Sending PUBLISH to gladys-main-instance-372063 (d0, q0, r0, m0, 'zigbee2mqtt/bridge/logging', ... (135 bytes))\\n\",\"stream\":\"stderr\",\"time\":\"2022-11-18T03:46:38.651112656Z\"}
{\"log\":\"1668743198: Sending PUBLISH to NodeRed-6dda341ee7a029cb (d0, q0, r0, m0, 'zigbee2mqtt/bridge/logging', ... (135 bytes))\\n\",\"stream\":\"stderr\",\"time\":\"2022-11-18T03:46:38.651137601Z\"}
{\"log\":\"1668743198: Sending PUBLISH to mqttjs_85adae46 (d0, q0, r0, m0, 'zigbee2mqtt/bridge/logging', ... (135 bytes))\\n\",\"stream\":\"stderr\",\"time\":\"2022-11-18T03:46:38.651298518Z\"}
{\"log\":\"1668743198: Received PUBLISH from mqttjs_85adae46 (d0, q0, r1, m0, 'zigbee2mqtt/capteurQualitéDeLair', ... (35 bytes))\\n\",\"stream\":\"stderr\",\"time\":\"2022-11-18T03:46:38.651313521Z\"}
{\"log\":\"1668743198: Sending PUBLISH to gladys-main-instance-362751 (d0, q0, r0, m0, 'zigbee2mqtt/capteurQualitéDeLair', ... (35 bytes))\\n\",\"stream\":\"stderr\",\"time\":\"2022-11-18T03:46:38.651318495Z\"}
{\"log\":\"1668743198: Sending PUBLISH to gladys-main-instance-372063 (d0, q0, r0, m0, 'zigbee2mqtt/capteurQualitéDeLair', ... (35 bytes))\\n\",\"stream\":\"stderr\",\"time\":\"2022-11-18T03:46:38.651322586Z\"}
{\"log\":\"1668743198: Sending PUBLISH to NodeRed-6dda341ee7a029cb (d0, q0, r0, m0, 'zigbee2mqtt/capteurQualitéDeLair', ... (35 bytes))\\n\",\"stream\":\"stderr\",\"time\":\"2022-11-18T03:46:38.651328966Z\"}
{\"log\":\"1668743198: Sending PUBLISH to mqttjs_85adae46 (d0, q0, r0, m0, 'zigbee2mqtt/capteurQualitéDeLair', ... (35 bytes))\\n\",\"stream\":\"stderr\",\"time\":\"2022-11-18T03:46:38.651354167Z\"}
{\"log\":\"1668743198: Received PUBLISH from mqttjs_85adae46 (d0, q0, r0, m0, 'zigbee2mqtt/bridge/logging', ... (283 bytes))\\n\",\"stream\":\"stderr\",\"time\":\"2022-11-18T03:46:38.693765258Z\"}
{\"log\":\"1668743198: Sending PUBLISH to gladys-main-instance-362751 (d0, q0, r0, m0, 'zigbee2mqtt/bridge/logging', ... (283 bytes))\\n\",\"stream\":\"stderr\",\"time\":\"2022-11-18T03:46:38.693813636Z\"}
{\"log\":\"1668743198: Sending PUBLISH to gladys-main-instance-372063 (d0, q0, r0, m0, 'zigbee2mqtt/bridge/logging', ... (283 bytes))\\n\",\"stream\":\"stderr\",\"time\":\"2022-11-18T03:46:38.693829505Z\"}
{\"log\":\"1668743198: Sending PUBLISH to NodeRed-6dda341ee7a029cb (d0, q0, r0, m0, 'zigbee2mqtt/bridge/logging', ... (283 bytes))\\n\",\"stream\":\"stderr\",\"time\":\"2022-11-18T03:46:38.69383429Z\"}
{\"log\":\"1668743198: Sending PUBLISH to mqttjs_85adae46 (d0, q0, r0, m0, 'zigbee2mqtt/bridge/logging', ... (283 bytes))\\n\",\"stream\":\"stderr\",\"time\":\"2022-11-18T03:46:38.693881662Z\"}
{\"log\":\"1668743198: Received PUBLISH from mqttjs_85adae46 (d0, q0, r0, m0, 'zigbee2mqtt/bridge/logging', ... (127 bytes))\\n\",\"stream\":\"stderr\",\"time\":\"2022-11-18T03:46:38.693935993Z\"}
{\"log\":\"1668743198: Sending PUBLISH to gladys-main-instance-362751 (d0, q0, r0, m0, 'zigbee2mqtt/bridge/logging', ... (127 bytes))\\n\",\"stream\":\"stderr\",\"time\":\"2022-11-18T03:46:38.693942686Z\"}
{\"log\":\"1668743198: Sending PUBLISH to gladys-main-instance-372063 (d0, q0, r0, m0, 'zigbee2mqtt/bridge/logging', ... (127 bytes))\\n\",\"stream\":\"stderr\",\"time\":\"2022-11-18T03:46:38.693996451Z\"}
{\"log\":\"1668743198: Sending PUBLISH to NodeRed-6dda341ee7a029cb (d0, q0, r0, m0, 'zigbee2mqtt/bridge/logging', ... (127 bytes))\\n\",\"stream\":\"stderr\",\"time\":\"2022-11-18T03:46:38.694010852Z\"}
{\"log\":\"1668743198: Sending PUBLISH to mqttjs_85adae46 (d0, q0, r0, m0, 'zigbee2mqtt/bridge/logging', ... (127 bytes))\\n\",\"stream\":\"stderr\",\"time\":\"2022-11-18T03:46:38.694042749Z\"}
{\"log\":\"1668743198: Received PUBLISH from mqttjs_85adae46 (d0, q0, r1, m0, 'zigbee2mqtt/capteurQualitéDeLair', ... (27 bytes))\\n\",\"stream\":\"stderr\",\"time\":\"2022-11-18T03:46:38.694107082Z\"}
{\"log\":\"1668743198: Sending PUBLISH to gladys-main-instance-362751 (d0, q0, r0, m0, 'zigbee2mqtt/capteurQualitéDeLair', ... (27 bytes))\\n\",\"stream\":\"stderr\",\"time\":\"2022-11-18T03:46:38.694133106Z\"}

Fix to consider?

Since this is probably a « bug », we should force the « emptying » of this file every day during Gladus backup, for example.

cicoub13 · January 11, 2023, 12:47pm

Thanks for the alert.
We could reduce the log_type to retain only warnings/errors.
I don’t see an option to reduce log retention, so the cleanup would have to be developed by us

VonOx · January 11, 2023, 2:11pm

On the Raspberry image I set a default setting in the Docker config

github.com/GladysAssistant/gladys-pi-gen

gladys/00-docker-install/files/daemon.json

master

{
  "log-driver": "json-file",
  "log-opts": {
    "max-size": "10m",
    "max-file": "3"
  }
}

I suggest you modify your Docker config, delete the log and restart the Docker daemon.

pierre-gilles · January 11, 2023, 7:45pm

Indeed the option "–log-opt max-size=10m \" is missing in the docker run of the MQTT integration (and maybe Zigbee2mqtt too), as we do with the Gladys container ( Installation avec Docker | Gladys Assistant )

It needs to be added, it’s a problem I agree!

Issue created:

AlexTrovato · January 14, 2023, 9:26am

I’ve made an initial fix for MQTT:

github.com/GladysAssistant/Gladys

Add MQTT container log limitation

master ← atrovato:mqtt-docker-limit-logs

ouvert 07:22PM - 13 Jan 23 UTC

atrovato

+25 -13

### Pull Request check-list To ensure your Pull Request can be accepted as fa…st as possible, make sure to review and check all of these items: - [x] If your changes affects code, did your write the tests? - [x] Are tests passing? (`npm test` on both front/server) - [x] Is the linter passing? (`npm run eslint` on both front/server) - [x] Did you run prettier? (`npm run prettier` on both front/server) - ~~[ ] If you are adding a new features/services, did you run integration comparator? (`npm run compare-translations` on front)~~ - [x] Did you test this pull request in real life? With real devices? If this development is a big feature or a new service, we recommend that you provide a Docker image to [the community](https://community.gladysassistant.com/) for testing before merging. - ~~[ ] If your changes modify the API (REST or Node.js), did you modify the API documentation? (Documentation is based on comments in code)~~ - ~~[ ] If you are adding a new features/services which needs explanation, did you modify the user documentation? See [the GitHub repo](https://github.com/GladysAssistant/v4-website) and the [website](https://gladysassistant.com).~~ - ~~[ ] Did you add fake requests data for the demo mode (`front/src/config/demo.js`) so that the demo website is working without a backend? (if needed) See [https://demo.gladysassistant.com](https://demo.gladysassistant.com).~~ NOTE: these things are not required to open a PR and can be done afterwards / while the PR is open. ### Description of change Add log limit on MQTT container. I add a new service variable to track the container update. Image : `atrovato/gladys:mqtt-log-limit`

pierre-gilles · January 20, 2023, 4:59am

Thanks @AlexTrovato for the PR, I tested it and it works fine on my side, I’ve merged

FYI, existing containers will be automatically restarted with the correct parameter on the next Gladys update, and this will automatically clear the existing logs.

AlexTrovato · January 21, 2023, 9:53am

@pierre-gilles I did the same for Zigbee, but I can’t test this one because it requires a Zigbee key to create the containers.

github.com/GladysAssistant/Gladys

Z2m docker limit logs

master ← atrovato:z2m-docker-limit-logs

ouvert 09:35AM - 21 Jan 23 UTC

atrovato

+225 -14

### Pull Request check-list To ensure your Pull Request can be accepted as fa…st as possible, make sure to review and check all of these items: - [x] If your changes affects code, did your write the tests? - [x] Are tests passing? (`npm test` on both front/server) - [x] Is the linter passing? (`npm run eslint` on both front/server) - [x] Did you run prettier? (`npm run prettier` on both front/server) - ~~[ ] If you are adding a new features/services, did you run integration comparator? (`npm run compare-translations` on front)~~ - [ ] Did you test this pull request in real life? With real devices? If this development is a big feature or a new service, we recommend that you provide a Docker image to [the community](https://community.gladysassistant.com/) for testing before merging. - ~~[ ] If your changes modify the API (REST or Node.js), did you modify the API documentation? (Documentation is based on comments in code)~~ - ~~[ ] If you are adding a new features/services which needs explanation, did you modify the user documentation? See [the GitHub repo]~~(https://github.com/GladysAssistant/v4-website) and the [website](https://gladysassistant.com). - ~~[ ] Did you add fake requests data for the demo mode (`front/src/config/demo.js`) so that the demo website is working without a backend? (if needed) See [https://demo.gladysassistant.com](https://demo.gladysassistant.com).~~ NOTE: these things are not required to open a PR and can be done afterwards / while the PR is open. ### Description of change Add log limit on Z2M containers. I add a new service variable to track the container update. Image : `atrovato/gladys:z2m-log-limit` ⚠️ I am not able to test it on my side due to lack of devices.

cicoub13 · January 21, 2023, 11:54am

Can you create a Docker image? I’ll run the test.

AlexTrovato · January 21, 2023, 12:43pm

You’ll find the image name in the PR description on GitHub ;)\nIt should have been generated since then.

pierre-gilles · January 23, 2023, 2:15am

Thanks @AlexTrovato! I’m waiting for feedback from @cicoub13 on the functional part

Would you like a Zigbee USB dongle, @AlexTrovato?

AlexTrovato · January 23, 2023, 7:21pm

Hi,
For development it’s true that it would be easier if I had a dongle and at least one sensor, especially since I’m considering other changes, like being able to connect to a remote z2m.
And that will lighten Vonox; he’ll be able to retrieve his environment for himself

lmilcent · January 23, 2023, 7:28pm

Or how to allow controlling two houses remotely in this case!

VonOx · January 23, 2023, 7:56pm

It was spare @AlexTrovato don’t worry

Tlse-vins · January 23, 2023, 7:58pm

I can send you a Zigbee2MQTT key; I have

pierre-gilles · January 24, 2023, 2:38am

So great, the remote Z2M

Thanks @Tlse-vins! Shall I let you sort out the key between you?

Regarding the devices, would you like one or several devices @AlexT

AlexTrovato · January 24, 2023, 6:15am

Ahah great, we’ll call her Mrs. Claus!!

I think an actuator like a smart plug and a temperature sensor (or multi-sensor) would make a good starter kit.
But if we have an idea for more complex integrations, it could be a good opportunity.

pierre-gilles · January 24, 2023, 6:26am

@AlexTrovato I suggest this for the plug:

https://www.domadoo.fr/fr/peripheriques/6165-nous-prise-intelligente-zigbee-30-mesure-de-consommation-5907772033517.html?domid=17

That way you’ll have the consumption measurement

For a temperature/humidity sensor, everything is out of stock at the moment; I had seen this one but it’s not available before the end of February:

https://www.domadoo.fr/fr/peripheriques/6197-nous-capteur-de-temperature-et-humidite-zigbee-30-tuya-5907772033890.html?domid=17

Otherwise I found a somewhat « fun » product that makes testing easy since you can click on it:

https://www.domadoo.fr/fr/peripheriques/5684-woox-telecommande-intelligente-4-boutons-zigbee-30-8435606701136.html?domid=17

I don’t think we support it either, so it could be nice

Otherwise you can look at this search if you see something useful: Domadoo sale — Zigbee sensors in stock

lmilcent · January 24, 2023, 6:51am

These sockets return a value every minute, or even more frequently depending on consumption. Also useful for getting continuous readings.

Tlse-vins · January 24, 2023, 9:05am

It’s a CC2652R ZZh that I have, if you want me to send it to you, give me your address in a private message.

VonOx · January 24, 2023, 5:48pm

That one is awesome!

Topic		Replies	Views
Améliorations dans l'UI de l'intégration zigbee2mqtt Développement	20	523	November 1, 2023
Point PR janvier 2023 Développement	19	800	January 20, 2023
Plusieurs erreurs/manques pour Tasmota POW-R2 Développement node-red	60	1913	July 3, 2023
Finir l'intégration du détecteur OWON PIR313-E [Archive] Demande de fonctionnalités	34	722	December 11, 2023
Gestion des volets roulants [Archive] Demande de fonctionnalités feature-integration	56	4105	October 4, 2022

Disk usage by Mosquitto logs

Fix to consider?

Related topics