Gladys crashes during GladysPlus backup

Configuration
1

I’ve been having trouble with my Gladys backup for several weeks, but I hadn’t taken the time to do the necessary checks.

In fact, every night when the backup is triggered, Gladys freezes (no dashboard display, no scene triggering, etc.), this lasts about 2 hours, then I receive the message « Gladys just restarted » in Telegram (this is the result of a scene with the trigger « Gladys starts »), and everything works again.

With Claude’s help this morning, I explored the issue. At one point in his analysis, I replied that I had « a medium-sized base (a few gigabytes) », but I’m not sure I answered correctly. In the history of my backups, I see this:

image
image1467×405 34.3 KB

As a first action, Claude suggested I modify the RAM allocated to Gladys (with the command sudo docker update --memory="6g" --memory-swap="8g" gladys), so that my issue no longer causes a freeze for several hours. And indeed, when I manually restarted a backup, the crash and restart took 3-4 minutes. It’s better to not block my home automation, but it doesn’t solve the backup failure issue yet :wink:

I’m running on an Intel NUC5PPYB mini-PC, N3700 processor 4 cores 1.6GHz, 8GB of RAM, under Ubuntu 22.04, and Gladys was installed with the standard « docker run » command. I don’t have anything else on this server besides Gladys.

Here is Claude’s final report:

Memory Peak / OOM-kill during GladysPlus backup

Summary

Every GladysPlus backup (automatic 2h-4h or manual) causes the Gladys container’s RAM to spike to the point of triggering an OOM-kill of the Node process, followed by an automatic restart of the container.

Environment

  • Gladys in Docker on Mini PC/NUC
  • Total host RAM: 7.7 GB (8 GB non-extensible)
  • Database: medium size (a few GB)
  • gladys container manually limited to 6 GB of RAM (docker update --memory) to prevent OOM from destabilizing the entire host (swap-thrashing observed before this limit was implemented)

Reproduction

Manual backup triggered at 09:36.

  • 09:36:01 - RSS before backup: 4388.77 MB, Heap 233.97/392.50 MB
  • 09:36:01 - DuckDB before backup: Total 2248.75 MB (BASE_TABLE: 2246.00 MB, IN_MEMORY_TABLE: 2.75 MB)
  • 09:36:04 - Opening a second DuckDB instance dedicated to backup (duckDbCreateBackupInstance, memory_limit=1024MB, threads=2, READ_ONLY) to export to a Parquet folder
  • 09:38 - Container RAM: ~6 GB (limit reached)
  • 09:40 - Out of memory: Killed process ... (MainThread)
  • 09:41 - Container restarted, Gladys functional again

Without the 6 GB limit manually set at the Docker level, the same phenomenon in real conditions (automatic nightly backup) caused a swap-thrashing episode on the entire host from 02:23 to 04:44 (dashboard and scenes unusable), the RSS of the process having reached 6774.54 MB before kill (dmesg: Out of memory: Killed process 1542 (MainThread) total-vm:30424440kB, anon-rss:6774536kB).

Hypothesis

The new DuckDB instance created for the backup (limited to 1024 MB) seems to be added to the memory already occupied by the main DuckDB instance (~2.25 GB), rather than sharing/respecting a global memory budget of the process. On a machine where the process is already running at ~4.4 GB at rest, adding ~1 GB more is enough to exceed the available RAM.

Questions for maintainers

  • Is it normal that BASE_TABLE remains almost entirely in memory (2.25 GB) at rest, outside of backup, for a medium-sized base?
  • Can the memory_limit=1024MB of the backup instance be lowered, or can the backup process stream/process in batches instead of loading so much in memory, to stay within a reasonable total envelope even on hardware with 8 GB of RAM?
2

@pierre-gilles I’m not sure you’ll have time to look at this before your vacation… It can wait for your return if it’s not possible :wink:

3

I’ll let @mutmut help you :slight_smile: (or someone else!)

4

@StephaneB
I have a bit of my troubles here: Migration intégration Netatmo : plantage de Gladys - #4 par mutmut
and then there: Surconsommation de mémoire par le backup de gateway? - #18 par mutmut

For now, the only solution I found to no longer be bothered is to increase the RAM :confused:
But as it’s a Proxmox cluster, if the host crashes, the LXC with Gladys will fail over to another host and then I risk having issues with shared RAM…
In any case, when it froze, I was able to unlock the situation by increasing the RAM live (thanks to LXC!), and all pending tasks completed successfully.

5

Thanks for the feedback @mutmut.

I just checked, my duckdb database is 8GB. And since I unfortunately can’t add more RAM (8GB max on this mini PC), I’m stuck…

Are there any services known to consume RAM that I could stop in the ‹ services › page of Gladys, just long enough to manually launch a backup?

6

Well, I managed to make a backup by shutting down all services except node-red and mqtt, as I wanted to continue monitoring RAM and CPU usage that I run in Node-Red (yes, I’ll have to switch to external integration…).

This reduced RAM usage by about -10%, dropping from around 80% to 70%. I then manually triggered a backup. Gladys couldn’t handle it, OOM crash. But on restart, RAM usage was only around 10%! So I immediately triggered a backup that went well.

In the process, I restarted all services and caused a reboot of the mini-PC. RAM usage stabilized at around 20%, great. It was 19:20.
But at 19:30, I had a CPU spike but especially a RAM usage increase to 60%, which hasn’t gone down since!

image
image1557×317 87.3 KB

Is it the energy tracking calculation that consumes a lot of RAM (why not) but ‹ doesn’t release it ›?

7

This isn’t a leak, it’s the DuckDB buffer pool filling up once and never being returned to the OS.

DuckDB is opened with memory_limit = 30% of RAM (models/index.js:214). Its buffer manager takes memory up to this limit on the first large scan and only evicts it under its own pressure, never to return RAM to the OS.
20% base + ~30% buffer pool + the Node heap that has grown ≈ the 60% observed. The 19:30 job is simply the first to perform heavy scans (getDeviceFeatureStates over a wide range after reboot, plus the DELETE + reinsertion of calculateCostFrom).

You can see how much DuckDB allocates on startup with this command:
docker logs gladys 2>&1 | grep "DuckDB initialized with memory_limit"

You can test a lower limit with the variable DUCKDB_MEMORY_LIMIT=512MB in your docker-compose. But I don’t know if this can have negative impacts on the rest.

But the second instance (the one that does the backup) may suffer from this non-eviction.
I think we should reduce the main pool during backup (to force eviction and leave room for the backup).
This is development work to be done

8

@pierre-gilles I don’t know if you’d have the opportunity to look at how to help me with this issue before your « life change », because I’m still quite bothered by the fact that Gladys Plus backups no longer work for me (except when I manually trigger them right after a Gladys restart).
I can’t say whether the solution suggested by @cicoub13 would be effective, nor if it’s a major core development or a small detail that Claude will handle in no time…

9

Hi everyone!

This topic is now in development.

A PR has been opened to reduce the DuckDB cache during Gladys Plus backup, to avoid OOM crashes:

Feel free to follow the PR, test (optional, especially for small requests) and give your feedback here if needed.

10

Hi @StephaneB,

I thought your issue was resolved: @cicoub13 had suggested a fix (lowering DUCKDB_MEMORY_LIMIT) and you hadn’t replied afterward, so I concluded it was fine on your end :slightly_smiling_face:

I looked into it closely, and @cicoub13’s analysis was correct:

  • DuckDB caches part of your database up to its memory limit. By default, it’s 30% of RAM, about 2.3 GB in your case: that’s your BASE_TABLE at 2.25 GB at rest. This cache is never released.
  • During backup, a second DuckDB instance opens for the export (limited to 1 GB), and its memory is added to this full cache. On a machine with 8 GB already running at 4.4 GB, this is enough to trigger the OOM.
  • That’s why a backup launched right after a restart works: the cache is still empty.

I made a PR that applies @cicoub13’s suggestion. During export, the main instance’s cache is reduced to 512 MB, then restored to its normal value at the end. This way, the backup memory no longer adds to a full cache:
#3190

Until the next version, you can work around the issue by limiting DuckDB’s cache to 1 GB. You need to recreate the container: docker stop gladys && docker rm gladys, then your usual docker run command with -e DUCKDB_MEMORY_LIMIT=1GB added.
To verify it’s taken into account:

docker logs gladys 2>&1 | grep "DuckDB initialized with memory_limit"

You can keep your 6 GB Docker limit as a safety net.

11

Thanks for the explanations, and especially thanks in advance for the correction!