Gladys container crash / memory full at night

Ok I made a build where I created a special « read-only » connection to the DB for the backup, which I close once the backup is finished.

In my tests, it seems to work but I’m curious to see your test :slight_smile:

It’s available on the same tag.

Ça a l’air pas mal !

2026-01-19T16:21:29+0100 <info> gateway.backup.js:35 (logMemoryUsage) JS memory before backup - RSS: 419.78 MB, Heap: 241.69/277.03 MB
2026-01-19T16:21:29+0100 <info> gateway.backup.js:41 (logMemoryUsage) DuckDB memory before backup - Total: 2.50 MB
2026-01-19T16:21:29+0100 <info> gateway.backup.js:46 ()   - BASE_TABLE: 1.00 MB
2026-01-19T16:21:29+0100 <info> gateway.backup.js:46 ()   - IN_MEMORY_TABLE: 1.50 MB
2026-01-19T16:21:29+0100 <info> system.isDocker.js:13 (System.isDocker) System.isDocker: This system is running inside Docker.
2026-01-19T16:21:29+0100 <info> gateway.backup.js:79 (Gateway.backup) Gateway backup: Locking SQLite Database
2026-01-19T16:21:29+0100 <info> gateway.backup.js:84 () Backup attempt n°1 : Cleaning backup folder
2026-01-19T16:21:29+0100 <info> gateway.backup.js:88 () Starting Gateway backup in folder /var/lib/gladysassistant/backups/gladys-db-backup-2026-1-19-16-21-29.db
2026-01-19T16:21:29+0100 <info> gateway.backup.js:90 () Gateway backup: Unlocking Database
2026-01-19T16:21:29+0100 <info> gateway.backup.js:96 (Gateway.backup) Gateway backup : SQLite file size is 2mb.
2026-01-19T16:21:29+0100 <info> gateway.backup.js:97 (Gateway.backup) Gateway backup : Backing up DuckDB into a Parquet folder /var/lib/gladysassistant/backups/gladys-db-backup_2026-1-19-16-21-29_parquet_folder
2026-01-19T16:22:33+0100 <info> gateway.backup.js:35 (logMemoryUsage) JS memory after export database to parquet - RSS: 2811.57 MB, Heap: 226.00/249.53 MB
2026-01-19T16:22:33+0100 <info> gateway.backup.js:41 (logMemoryUsage) DuckDB memory after export database to parquet - Total: 2.75 MB
2026-01-19T16:22:33+0100 <info> gateway.backup.js:46 ()   - BASE_TABLE: 1.00 MB
2026-01-19T16:22:33+0100 <info> gateway.backup.js:46 ()   - IN_MEMORY_TABLE: 1.75 MB
2026-01-19T16:22:33+0100 <info> gateway.backup.js:111 (Gateway.backup) Gateway backup: Closing DuckDB backup instance to release memory
2026-01-19T16:22:33+0100 <info> gateway.backup.js:35 (logMemoryUsage) JS memory after closing backup instance - RSS: 2811.45 MB, Heap: 226.47/249.53 MB
2026-01-19T16:22:33+0100 <info> gateway.backup.js:41 (logMemoryUsage) DuckDB memory after closing backup instance - Total: 2.75 MB
2026-01-19T16:22:33+0100 <info> gateway.backup.js:46 ()   - BASE_TABLE: 1.00 MB
2026-01-19T16:22:33+0100 <info> gateway.backup.js:46 ()   - IN_MEMORY_TABLE: 1.75 MB
2026-01-19T16:22:33+0100 <info> gateway.backup.js:116 (Gateway.backup) Gateway backup: Compressing backup
2026-01-19T16:23:01+0100 <info> gateway.backup.js:122 (Gateway.backup) Gateway backup: Encrypting backup
2026-01-19T16:23:03+0100 <info> gateway.backup.js:129 (Gateway.backup) Gateway backup: Uploading backup, size of encrypted backup = 443mb. Path = /var/lib/gladysassistant/backups/gladys-db-backup-2026-1-19-16-21-29.tar.gz.enc
2026-01-19T16:23:03+0100 <info> gateway.backup.js:35 (logMemoryUsage) JS memory before upload chunk 0 - RSS: 2614.45 MB, Heap: 232.99/256.53 MB
2026-01-19T16:23:03+0100 <info> gateway.backup.js:41 (logMemoryUsage) DuckDB memory before upload chunk 0 - Total: 2.75 MB
2026-01-19T16:23:03+0100 <info> gateway.backup.js:46 ()   - BASE_TABLE: 1.00 MB
2026-01-19T16:23:03+0100 <info> gateway.backup.js:46 ()   - IN_MEMORY_TABLE: 1.75 MB
2026-01-19T16:23:04+0100 <info> gateway.backup.js:35 (logMemoryUsage) JS memory before upload chunk 1 - RSS: 444.46 MB, Heap: 221.74/259.78 MB
2026-01-19T16:23:04+0100 <info> gateway.backup.js:41 (logMemoryUsage) DuckDB memory before upload chunk 1 - Total: 2.75 MB
2026-01-19T16:23:04+0100 <info> gateway.backup.js:46 ()   - BASE_TABLE: 1.00 MB
2026-01-19T16:23:04+0100 <info> gateway.backup.js:46 ()   - IN_MEMORY_TABLE: 1.75 MB
2026-01-19T16:23:05+0100 <info> gateway.backup.js:35 (logMemoryUsage) JS memory before upload chunk 2 - RSS: 417.40 MB, Heap: 210.64/254.78 MB
2026-01-19T16:23:05+0100 <info> gateway.backup.js:41 (logMemoryUsage) DuckDB memory before upload chunk 2 - Total: 2.75 MB
2026-01-19T16:23:05+0100 <info> gateway.backup.js:46 ()   - BASE_TABLE: 1.00 MB
2026-01-19T16:23:05+0100 <info> gateway.backup.js:46 ()   - IN_MEMORY_TABLE: 1.75 MB
2026-01-19T16:23:06+0100 <info> gateway.backup.js:35 (logMemoryUsage) JS memory before upload chunk 3 - RSS: 432.16 MB, Heap: 211.50/246.59 MB
2026-01-19T16:23:06+0100 <info> gateway.backup.js:41 (logMemoryUsage) DuckDB memory before upload chunk 3 - Total: 2.75 MB
2026-01-19T16:23:06+0100 <info> gateway.backup.js:46 ()   - BASE_TABLE: 1.00 MB
2026-01-19T16:23:06+0100 <info> gateway.backup.js:46 ()   - IN_MEMORY_TABLE: 1.75 MB
2026-01-19T16:23:06+0100 <info> gateway.backup.js:35 (logMemoryUsage) JS memory before upload chunk 4 - RSS: 451.83 MB, Heap: 212.45/246.59 MB
2026-01-19T16:23:06+0100 <info> gateway.backup.js:41 (logMemoryUsage) DuckDB memory before upload chunk 4 - Total: 2.75 MB
2026-01-19T16:23:06+0100 <info> gateway.backup.js:46 ()   - BASE_TABLE: 1.00 MB
2026-01-19T16:23:06+0100 <info> gateway.backup.js:46 ()   - IN_MEMORY_TABLE: 1.75 MB
2026-01-19T16:23:07+0100 <info> gateway.backup.js:35 (logMemoryUsage) JS memory before upload chunk 5 - RSS: 472.02 MB, Heap: 207.63/246.59 MB
2026-01-19T16:23:07+0100 <info> gateway.backup.js:41 (logMemoryUsage) DuckDB memory before upload chunk 5 - Total: 2.75 MB
2026-01-19T16:23:07+0100 <info> gateway.backup.js:46 ()   - BASE_TABLE: 1.00 MB
2026-01-19T16:23:07+0100 <info> gateway.backup.js:46 ()   - IN_MEMORY_TABLE: 1.75 MB
2026-01-19T16:23:08+0100 <info> gateway.backup.js:35 (logMemoryUsage) JS memory before upload chunk 6 - RSS: 492.15 MB, Heap: 208.47/246.59 MB
2026-01-19T16:23:08+0100 <info> gateway.backup.js:41 (logMemoryUsage) DuckDB memory before upload chunk 6 - Total: 2.75 MB
2026-01-19T16:23:08+0100 <info> gateway.backup.js:46 ()   - BASE_TABLE: 1.00 MB
2026-01-19T16:23:08+0100 <info> gateway.backup.js:46 ()   - IN_MEMORY_TABLE: 1.75 MB
2026-01-19T16:23:08+0100 <info> gateway.backup.js:35 (logMemoryUsage) JS memory before upload chunk 7 - RSS: 471.50 MB, Heap: 207.71/246.84 MB
2026-01-19T16:23:08+0100 <info> gateway.backup.js:41 (logMemoryUsage) DuckDB memory before upload chunk 7 - Total: 2.75 MB
2026-01-19T16:23:08+0100 <info> gateway.backup.js:46 ()   - BASE_TABLE: 1.00 MB
2026-01-19T16:23:08+0100 <info> gateway.backup.js:46 ()   - IN_MEMORY_TABLE: 1.75 MB
2026-01-19T16:23:09+0100 <info> gateway.backup.js:35 (logMemoryUsage) JS memory before upload chunk 8 - RSS: 491.62 MB, Heap: 208.85/246.84 MB
2026-01-19T16:23:09+0100 <info> gateway.backup.js:41 (logMemoryUsage) DuckDB memory before upload chunk 8 - Total: 2.75 MB
2026-01-19T16:23:09+0100 <info> gateway.backup.js:46 ()   - BASE_TABLE: 1.00 MB
2026-01-19T16:23:09+0100 <info> gateway.backup.js:46 ()   - IN_MEMORY_TABLE: 1.75 MB
2026-01-19T16:23:10+0100 <info> gateway.backup.js:35 (logMemoryUsage) JS memory before upload chunk 9 - RSS: 472.57 MB, Heap: 208.25/246.84 MB
2026-01-19T16:23:10+0100 <info> gateway.backup.js:41 (logMemoryUsage) DuckDB memory before upload chunk 9 - Total: 2.75 MB
2026-01-19T16:23:10+0100 <info> gateway.backup.js:46 ()   - BASE_TABLE: 1.00 MB
2026-01-19T16:23:10+0100 <info> gateway.backup.js:46 ()   - IN_MEMORY_TABLE: 1.75 MB
2026-01-19T16:23:11+0100 <info> gateway.backup.js:35 (logMemoryUsage) JS memory before upload chunk 10 - RSS: 492.07 MB, Heap: 209.87/246.84 MB
2026-01-19T16:23:11+0100 <info> gateway.backup.js:41 (logMemoryUsage) DuckDB memory before upload chunk 10 - Total: 2.75 MB
2026-01-19T16:23:11+0100 <info> gateway.backup.js:46 ()   - BASE_TABLE: 1.00 MB
2026-01-19T16:23:11+0100 <info> gateway.backup.js:46 ()   - IN_MEMORY_TABLE: 1.75 MB
2026-01-19T16:23:12+0100 <info> gateway.backup.js:35 (logMemoryUsage) JS memory before upload chunk 11 - RSS: 474.23 MB, Heap: 211.14/246.59 MB
2026-01-19T16:23:12+0100 <info> gateway.backup.js:41 (logMemoryUsage) DuckDB memory before upload chunk 11 - Total: 2.75 MB
2026-01-19T16:23:12+0100 <info> gateway.backup.js:46 ()   - BASE_TABLE: 1.00 MB
2026-01-19T16:23:12+0100 <info> gateway.backup.js:46 ()   - IN_MEMORY_TABLE: 1.75 MB
2026-01-19T16:23:13+0100 <info> gateway.backup.js:35 (logMemoryUsage) JS memory before upload chunk 12 - RSS: 494.98 MB, Heap: 207.42/246.59 MB
2026-01-19T16:23:13+0100 <info> gateway.backup.js:41 (logMemoryUsage) DuckDB memory before upload chunk 12 - Total: 2.75 MB
2026-01-19T16:23:13+0100 <info> gateway.backup.js:46 ()   - BASE_TABLE: 1.00 MB
2026-01-19T16:23:13+0100 <info> gateway.backup.js:46 ()   - IN_MEMORY_TABLE: 1.75 MB
2026-01-19T16:23:14+0100 <info> gateway.backup.js:35 (logMemoryUsage) JS memory before upload chunk 13 - RSS: 455.16 MB, Heap: 207.80/246.59 MB
2026-01-19T16:23:14+0100 <info> gateway.backup.js:41 (logMemoryUsage) DuckDB memory before upload chunk 13 - Total: 2.75 MB
2026-01-19T16:23:14+0100 <info> gateway.backup.js:46 ()   - BASE_TABLE: 1.00 MB
2026-01-19T16:23:14+0100 <info> gateway.backup.js:46 ()   - IN_MEMORY_TABLE: 1.75 MB
2026-01-19T16:23:15+0100 <info> gateway.backup.js:35 (logMemoryUsage) JS memory before upload chunk 14 - RSS: 495.15 MB, Heap: 209.56/246.59 MB
2026-01-19T16:23:15+0100 <info> gateway.backup.js:41 (logMemoryUsage) DuckDB memory before upload chunk 14 - Total: 2.75 MB
2026-01-19T16:23:15+0100 <info> gateway.backup.js:46 ()   - BASE_TABLE: 1.00 MB
2026-01-19T16:23:15+0100 <info> gateway.backup.js:46 ()   - IN_MEMORY_TABLE: 1.75 MB
2026-01-19T16:23:16+0100 <info> gateway.backup.js:35 (logMemoryUsage) JS memory before upload chunk 15 - RSS: 474.92 MB, Heap: 209.01/246.84 MB
2026-01-19T16:23:16+0100 <info> gateway.backup.js:41 (logMemoryUsage) DuckDB memory before upload chunk 15 - Total: 2.75 MB
2026-01-19T16:23:16+0100 <info> gateway.backup.js:46 ()   - BASE_TABLE: 1.00 MB
2026-01-19T16:23:16+0100 <info> gateway.backup.js:46 ()   - IN_MEMORY_TABLE: 1.75 MB
2026-01-19T16:23:17+0100 <info> gateway.backup.js:35 (logMemoryUsage) JS memory before upload chunk 16 - RSS: 494.77 MB, Heap: 210.85/246.84 MB
2026-01-19T16:23:17+0100 <info> gateway.backup.js:41 (logMemoryUsage) DuckDB memory before upload chunk 16 - Total: 2.75 MB
2026-01-19T16:23:17+0100 <info> gateway.backup.js:46 ()   - BASE_TABLE: 1.00 MB
2026-01-19T16:23:17+0100 <info> gateway.backup.js:46 ()   - IN_MEMORY_TABLE: 1.75 MB
2026-01-19T16:23:19+0100 <info> gateway.backup.js:35 (logMemoryUsage) JS memory before upload chunk 17 - RSS: 454.84 MB, Heap: 211.07/246.84 MB
2026-01-19T16:23:19+0100 <info> gateway.backup.js:41 (logMemoryUsage) DuckDB memory before upload chunk 17 - Total: 2.75 MB
2026-01-19T16:23:19+0100 <info> gateway.backup.js:46 ()   - BASE_TABLE: 1.00 MB
2026-01-19T16:23:19+0100 <info> gateway.backup.js:46 ()   - IN_MEMORY_TABLE: 1.75 MB
2026-01-19T16:23:21+0100 <info> gateway.backup.js:35 (logMemoryUsage) JS memory before upload chunk 18 - RSS: 495.13 MB, Heap: 207.39/246.84 MB
2026-01-19T16:23:21+0100 <info> gateway.backup.js:41 (logMemoryUsage) DuckDB memory before upload chunk 18 - Total: 2.75 MB
2026-01-19T16:23:21+0100 <info> gateway.backup.js:46 ()   - BASE_TABLE: 1.00 MB
2026-01-19T16:23:21+0100 <info> gateway.backup.js:46 ()   - IN_MEMORY_TABLE: 1.75 MB
2026-01-19T16:23:23+0100 <info> gateway.backup.js:35 (logMemoryUsage) JS memory before upload chunk 19 - RSS: 495.12 MB, Heap: 209.18/246.84 MB
2026-01-19T16:23:23+0100 <info> gateway.backup.js:41 (logMemoryUsage) DuckDB memory before upload chunk 19 - Total: 2.75 MB
2026-01-19T16:23:23+0100 <info> gateway.backup.js:46 ()   - BASE_TABLE: 1.00 MB
2026-01-19T16:23:23+0100 <info> gateway.backup.js:46 ()   - IN_MEMORY_TABLE: 1.75 MB
2026-01-19T16:23:25+0100 <info> gateway.backup.js:35 (logMemoryUsage) JS memory before upload chunk 20 - RSS: 474.77 MB, Heap: 207.70/246.59 MB
2026-01-19T16:23:25+0100 <info> gateway.backup.js:41 (logMemoryUsage) DuckDB memory before upload chunk 20 - Total: 2.75 MB
2026-01-19T16:23:25+0100 <info> gateway.backup.js:46 ()   - BASE_TABLE: 1.00 MB
2026-01-19T16:23:25+0100 <info> gateway.backup.js:46 ()   - IN_MEMORY_TABLE: 1.75 MB
2026-01-19T16:23:27+0100 <info> gateway.backup.js:35 (logMemoryUsage) JS memory before upload chunk 21 - RSS: 494.90 MB, Heap: 209.08/246.59 MB
2026-01-19T16:23:27+0100 <info> gateway.backup.js:41 (logMemoryUsage) DuckDB memory before upload chunk 21 - Total: 2.75 MB
2026-01-19T16:23:27+0100 <info> gateway.backup.js:46 ()   - BASE_TABLE: 1.00 MB
2026-01-19T16:23:27+0100 <info> gateway.backup.js:46 ()   - IN_MEMORY_TABLE: 1.75 MB
2026-01-19T16:23:28+0100 <info> gateway.backup.js:35 (logMemoryUsage) JS memory before upload chunk 22 - RSS: 454.86 MB, Heap: 210.96/246.84 MB
2026-01-19T16:23:28+0100 <info> gateway.backup.js:41 (logMemoryUsage) DuckDB memory before upload chunk 22 - Total: 2.75 MB
2026-01-19T16:23:28+0100 <info> gateway.backup.js:46 ()   - BASE_TABLE: 1.00 MB
2026-01-19T16:23:28+0100 <info> gateway.backup.js:46 ()   - IN_MEMORY_TABLE: 1.75 MB
2026-01-19T16:23:29+0100 <info> gateway.backup.js:175 (Gateway.backup) Gladys backup uploaded with success to Gladys Gateway.
2026-01-19T16:23:29+0100 <info> gateway.backup.js:35 (logMemoryUsage) JS memory after upload - RSS: 475.01 MB, Heap: 213.18/246.84 MB
2026-01-19T16:23:29+0100 <info> gateway.backup.js:41 (logMemoryUsage) DuckDB memory after upload - Total: 2.75 MB
2026-01-19T16:23:29+0100 <info> gateway.backup.js:46 ()   - BASE_TABLE: 1.00 MB
2026-01-19T16:23:29+0100 <info> gateway.backup.js:46 ()   - IN_MEMORY_TABLE: 1.75 MB

image

Nice! I’ll clean up the PR and integrate that into Gladys then :slight_smile:

1 Like

I’m reverting to the latest production version and I’ll continue to monitor whether my other underlying issue reappears.

That way, I’m trying to provide some details.

1 Like

The fix is live in Gladys Assistant v4.67 :

1 Like

Hello @pierre-gilles,

For my part the problem is still present.
This morning nothing was working, I checked and Gladys is down

image

As I’m out of RAM, it’s impossible to connect to view the logs.
Fortunately I’m running on Proxmox and can change the RAM on the fly.
I bumped it up to 13GB and Gladys came back up.

And the RAM usage dropped
image

It should stabilize around 6GB.

When Gladys came back up, I was able to look at the logs; there’s the end of the backup I think and everything related to energy consumption.

2026-01-24T09:25:35+0100 \u003cinfo\u003e energy-monitoring.calculateConsumptionFromIndex.js:160 (EnergyMonitoringHandler.calculateConsumptionFromIndex) Finished calculating consumption from index for 12 devices
2026-01-24T09:25:35+0100 \u003cinfo\u003e energy-monitoring.calculateConsumptionFromIndex.js:27 (EnergyMonitoringHandler.calculateConsumptionFromIndex) Calculating consumption from index in timezone Europe/Paris for window Sat Jan 24 2026 09:00:00 GMT+0100 (Central European Standard Time)
2026-01-24T09:25:35+0100 \u003cinfo\u003e energy-monitoring.calculateConsumptionFromIndex.js:34 (EnergyMonitoringHandler.calculateConsumptionFromIndex) Found 76 energy devices
2026-01-24T09:25:35+0100 \u003cinfo\u003e energy-monitoring.calculateConsumptionFromIndex.js:65 (EnergyMonitoringHandler.calculateConsumptionFromIndex) Found 12 devices with both INDEX and THIRTY_MINUTES_CONSUMPTION features
2026-01-24T09:25:36+0100 \u003cinfo\u003e gateway.backup.js:110 (Gateway.backup) Gateway backup: Closing DuckDB backup instance to release memory
2026-01-24T09:25:36+0100 \u003cinfo\u003e gateway.backup.js:114 (Gateway.backup) Gateway backup: Compressing backup

But I don’t have any specific error between the moment Gladys froze and the moment Gladys came back up. With Gladys stuck nothing was happening…

It’s really annoying nonetheless, especially when my wife isn’t happy first thing in the morning when nothing works… :grin:

Edit: FYI I switched to version 4.67 yesterday when you released the new version.

Hi @_Will_71,

I don’t remember if I was talking about this with you, but if you’re on Proxmox I think I understand what’s happening.

Gladys detects the available RAM on the system at startup and allocates up to 30% of that amount to DuckDB.

However, on Proxmox, it’s possible that the RAM exposed to Gladys is the system’s total RAM and not only the RAM actually allocated to Gladys.

If you want to check this setting, at Gladys startup you have a log:

2026-01-23T16:02:47+0100 <info> index.js:101 (Object.<anonymous>) DuckDB initialized with memory_limit=4730MB (system RAM: 15767MB)
2026-01-23T16:02:49+0100 <info> index.js:163 (Object.duckDbShowVersion) DuckDB version = v1.4.3
2026-01-23T16:02:49+0100 <info> index.js:167 (Object.duckDbShowVersion) DuckDB memory_limit = 4.4 GiB

Can you check on your side what this log shows?

For me, with Proxmox in VM mode I don’t encounter this type of problem even if the RAM is a bit loaded:
image

@prohand Can you check the log when Gladys starts?

I have this on my end:

2026-01-24T10:21:57+0100 \u003cinfo\u003e index.js:101 (Object.\u003canonymous\u003e) DuckDB initialized with memory_limit=1173MB (system RAM: 3911MB)

Ok, everything is good on your side @prohand :slight_smile:

By the way, the « Memory usage » line in Proxmox is technically wrong — htop is correct. Only 612 MB are actually used on your system. Proxmox includes the « buffer/cache » which are

1 Like

[quote=« pierre-gilles, post:48, topic:9895 »]
Gladys detects at startup the RAM available on the system and allocates a maximum of 30% of that amount to Duck

1 Like

I can no longer reproduce the issue on my end :sweat_smile:

We’ll see if it reappears one day, but it was something that showed up very easily and the only thing I actually did while debugging was a big cleanup of the DuckDB data. Even though I then restored the database to its pre-cleanup state, for now, no more issues.

1 Like

There you go

2026-01-24T12:27:18+0100 \u003cinfo\u003e index.js:101 (Object.\u003canonymous\u003e) DuckDB initialized with memory_limit=9596MB (system RAM: 31989MB)
2026-01-24T12:27:45+

Ok, I might have a lead to find the available RAM for Gladys in the case of Proxmox/LXC.

Could you run (taking into account that « gladys » is the name of your Docker container, change it if needed):

docker exec -it gladys cat /sys/fs/cgroup/memory.max

and:

docker exec -it gladys cat /sys/fs/cgroup/memory/memory.limit_in_bytes

And show me here the output of these commands?

Here it is:

image

Below is what I have in /sys/fs/cgroup/
image
image

Ok, can you run:

docker exec -it gladys cat /proc/meminfo | grep MemTotal

image

This is crazy, there’s no way to know what RAM is allocated, Proxmox exposes the system’s total RAM everywhere…

In the meantime, you can set the memory limit yourself by adding to the docker run:

-e DUCKDB_MEMORY_LIMIT=4000MB

With 12 GB available, you can give 4 GB to DuckDB, I think.

But I’d really like us to find a proper automatic solution, because that’s not a real solution…

Yes, that’s what I was planning to do.
Thanks for your feedback.