Zigbee2mqtt: Docker test image based on Gladys v4

Yes, I will change the log level.

Why are you making 400 requests per minute? Do you have hundreds of devices?

In the range, it’s stylish indeed! :slight_smile:

I think I know where this comes from. In fact, every time a new sensor value is published, Gladys must check that no scenes need to be triggered.

Currently, what is done is that for each sensor value received, Gladys goes through each existing scene and checks whether the scene validates or not the condition. It’s a simple synchronous forEach which is therefore blocking.

In terms of complexity, it’s O(n). The more scenes there are to go through, the longer it takes. Especially in your case where you have 400 requests/minute (6.6 requests/sec), that’s a lot of loop iterations :smiley: Out of curiosity, how many scenes do you have?

I had this limitation in mind, and I had created a GitHub issue to fix it: Each Node.js event should trigger a O(1) function, not O(n) · Issue #803 · GladysAssistant/Gladys · GitHub

Basically, the idea would be to find a way to store the scenes differently in RAM, in a format where we would be able to check that a scene is valid in a simple 0(1), at least for events of type « device new state »

I use the Zigbee network to create ambilight with my lamps, and as a result, it sends a lot of requests :sweat_smile: I even managed to crash my Zigbee network.

So, I reduced the number of requests, but Gladys still has trouble.

I currently only have one scene which is a scheduled trigger.

I wondered if it wasn’t because Gladys records each new value in the database?

Are you using an SSD or an SD card?

Indeed, Gladys records values in the database, and if you’re using an SD card, it can become the limiting factor.

In general, I really recommend SSDs for a long-term installation.

The Raspberry OS runs on an SD card, but Gladys’ data is saved on a hard drive connected via USB3 on the Raspberry.

A hard drive with platters? You should check the disk speed, but indeed if you continuously try to write at 6 req/second + read, the requests will be put on hold and you will see slowdowns on Gladys’ side

How did you set up your ambilight?

Indeed, I don’t think it’s a good idea to record data every time the state changes, plus your DB will become huge!

Yes, the good old hard drives :grin:

I have an Express server that emulates the Philips Hue API and then sends MQTT to Zigbee. To transmit the screen, I use Hyperion NG.

I think so too, let’s say, I leave it running for 1 hour, I can’t even imagine the number of records made :no_mouth:

And so Gladys retrieves the events because Zigbee2mqtt returns the value feedback to Gladys?

Yes, it seems that Gladys listens to the devices registered on the Zigbee2mqtt integration (such as the Hue Play device).

As a result, all values received by Zigbee and recorded on Gladys are then received by Gladys.

The path being:

Hypérion → Api Philips (Express) → Zigbee2Mqtt → Device && Gladys

Oh, I see!

First, are you sure it’s best practice to hit the Philips Hue API so hard? (in terms of bulb lifespan, Zigbee network, etc.?)

I don’t know Zigbee2mqtt well, I don’t know if it’s possible to disable event sending to Gladys on the Zigbee2mqtt side, I’ll let @cicoub13 answer that.

Well…in fact, I just recreated what Philips does through their Hue Sync software, which allows you to cast the lights from your screen onto lamps. We’ll see how it holds up over time.

I think it would be better to add a button to disable the recording of old values.

Or yes, disable the sending of events, but I think we should still be able to communicate with Zigbee2mqtt in both directions.

We already have in Gladys the ability not to record the history of a sensor (it’s not in the Zigbee2mqtt service UI but it’s a feature that is present in Gladys core), but that won’t change anything in your case, since we always record the last value (to be able to display the last value on the dashboard simply).

If we leave the possibility to stop writing all values, when you display your dashboard the displayed value will be wrong.

Indeed, there will always be part of the problem, but at least the database won’t be overloaded, right?

Indeed!

We should add a toggle « Save value history » in the Zigbee2mqtt service UI, which corresponds to the « keep_history » attribute in the « device_feature » table.

See code: Gladys/server/models/device_feature.js at master · GladysAssistant/Gladys · GitHub

Do you want to take on the development @JeuFore? :slight_smile:

@cicoub13 can help you understand the structure of the Zigbee2mqtt integration if you need help, he’s the one who worked the most on it :slight_smile:

@cicoub13 I had a power outage

After the restart, no more z2m / mqtt

2021-06-22T14:20:11+0200 <warn> service.start.js:44 (Service.start) Unable to start service zigbee2mqtt Error: (HTTP code 301) unexpected -
    at /src/server/node_modules/docker-modem/lib/modem.js:301:17
    at getCause (/src/server/node_modules/docker-modem/lib/modem.js:331:7)
    at Modem.buildPayload (/src/server/node_modules/docker-modem/lib/modem.js:300:5)
    at IncomingMessage.<anonymous> (/src/server/node_modules/docker-modem/lib/modem.js:275:14)
    at IncomingMessage.emit (events.js:388:22)
    at endReadableNT (internal/streams/readable.js:1336:12)
    at processTicksAndRejections (internal/process/task_queues.js:82:21) {
  reason: undefined,
  statusCode: 301,
  json: <Buffer >
}
2021-06-22T14:20:11+0200 <error> index.js:20 (process.<anonymous>) uncaughtException catched: uncaughtException
2021-06-22T14:20:11+0200 <error> index.js:21 (process.<anonymous>) Error: getaddrinfo ENOTFOUND containers
    at GetAddrInfoReqWrap.onlookup [as oncomplete] (dns.js:67:26) {
  errno: -3008,
  code: 'ENOTFOUND',
  syscall: 'getaddrinfo',
  hostname: 'containers'
}

Everything is okay with Docker

Home Assistant has no connection issues with the same MQTT server.

I don’t understand why I ended up in this unstable state.

I noticed this because the « Temperature in room » box had no recent data.

Unable to configure the service (the dongle is configured)

Lots of information, but if you need anything else, don’t hesitate.


EDIT: I had to restart my server (physical), it fixed my issue, it’s probably the dockerode library that messed up.

I should be able to free up some time in the coming days, so why not take a closer look at it :slight_smile:

Hello,

So, I started looking into the keep_history feature on Zigbee2mqtt.

I’ll test it this weekend to see if it works in real conditions, as I’m not at home right now.

Hello. The temperature of the bulbs is now managed (with a nice slider).
Can you update the image cicoub13/gladys:dev-zigbee2mqtt and test again (spoiler, I tested and it works great)? It should arrive soon in the version available to everyone :rocket:

Hello,

Okay, I’m testing the image this weekend. By the way, a quick question, does anyone have problems creating Zigbee containers via Gladys :thinking:?