Is the gateway backup consuming too much memory?

Configuration
1

Hi,

I don’t know if I’m the only one, but I regularly get OOM errors when I allocate 8Gb to Gladys. This causes Gladys to restart every night.

After days of debugging without understanding the cause, I used Claude to move forward and identify a better hypothesis. And in the following days the problem was identified: the way the database is backed up.

Context: I keep all states indefinitely, as long as I have disk space.

Am I the only one this happens to?

Bug: the backup Gateway can cause an OOM-kill of the Node process

Components affected:
server/lib/gateway/gateway.backup.js
server/models/index.js (function duckDbCreateBackupInstance)

The problem: duckDbCreateBackupInstance() opens a new separate DuckDB instance on the same .duckdb file as the main instance (already active in read/write):

const backupDatabase = await DuckDBInstance.create(duckDbFilePath, {
  memory_limit: duckDbMemoryLimit,
  access_mode: 'READ_ONLY',
});

However, the official documentation of the Node.js « Neo » client for DuckDB explicitly states:

« Multiple instances in the same process should not attach the same database. »

The mechanism intended to avoid this is DuckDBInstanceCache.getOrCreateInstance(path), which reuses the same instance (and thus the same buffer pool / memory_limit) for the same file. The current code directly calls DuckDBInstance.create(), which creates two independent buffer pools, each capped at the same DUCKDB_MEMORY_LIMIT.

Result: the worst-case memory usage of DuckDB alone becomes 2 × memory_limit instead of 1 × memory_limit.

Worsening: the DuckDB documentation also specifies that memory_limit « only applies to the buffer manager » — compression buffers (GZIP compression used in EXPORT DATABASE) can consume in addition to this limit.

Proof in production (2 identical crashes, logs included):

  • RSS stable ~2.5Gb before backup (including ~1.2Gb of DuckDB BASE_TABLE)
  • Jump to ~4.6Gb in less than a minute, right after the log « Backing up DuckDB into a Parquet folder »
  • Process killed (ExitCode=137) before reaching the log « Closing DuckDB backup instance to release memory » in the finally — so the crash occurs during the EXPORT DATABASE … FORMAT PARQUET COMPRESSION GZIP call itself, not after.
  • Reproduced identically for 2 days in a row, always at the time of the daily Gateway backup.

What is requested:

  1. Make sure that duckDbCreateBackupInstance() uses DuckDBInstanceCache.getOrCreateInstance() instead of DuckDBInstance.create(), so that the backup shares the buffer pool of the main instance rather than opening a second one.
  2. Consider replacing COMPRESSION GZIP with ZSTD or SNAPPY in the EXPORT DATABASE (less compression buffers, faster export).
  3. Possibly document that DUCKDB_MEMORY_LIMIT should be sized anticipating that two instances may coexist during a backup (so recommend a value ≤ 25% of the container’s allocated RAM, not 30-50%), as long as point 1 is not fixed.

Context to situate the impact: installation with ~8Gb allocated to the container, BASE_TABLE of about 1.1-1.2Gb of sensor history — so not an extreme case, this behavior can probably affect other installations with a significant history.

2

Well, it turns out I’ve had the same issue for 7 days!
However, I have to restart my LXC Docker with Gladys every morning because nothing responds anymore.

I’m working with Claude, who is supposed to create a watchdog and an automatic restart after a crash, with log recovery just before the OOM because after that, I can’t access anything anymore.

I’ll keep you updated.

3

Thanks for the feedback, I’ll check it out!

4

For the first time this morning, it didn’t crash, no OOM:

image
image683×627 55.7 KB

image
image668×376 23.2 KB

image
image661×493 37.4 KB

And I haven’t set up the script yet :frowning:

5

Since I modified my Docker Compose to limit DuckDB to a maximum of 2GB instead of 4GB, no more crashes (at least not last night). This confirms that I’m on the right track.

6

Mine was already limited to 2GB since the first OOM issues.
When it started again a week ago, I even limited it to 1.5GB but without success.

version: '3'

services:
  gladys:
    image: gladysassistant/gladys:v4
    container_name: gladys
    restart: always
    privileged: true
    network_mode: host
    cgroup: host
    logging:
      driver: "json-file"
      options:
        max-size: 10m
    environment:
      NODE_ENV: production
      SQLITE_FILE_PATH: /var/lib/gladysassistant/gladys-production.db
      SERVER_PORT: 8420
      TZ: Europe/Paris
      DUCKDB_MEMORY_LIMIT: 1500MB # memory usage limitation for duckDB
7

Normally @pierre-gilles has already set up an automatic limitation of xx% for duckDB (I don’t remember the exact figure), which works very well except for installations with proxmox where the RAM value of the LXC is not correctly retrieved by docker, and in this case it is indeed necessary to limit (unless LXC RAM = host RAM).

8

That’s exactly it, LXC with 8GiB, Docker seeing the server’s 64GiB, and OOM.

This is clearly not a « mainstream » issue, but it has already happened several times, I see!

9

Hi @lmilcent,

Indeed, there are 3 distinct phenomena here :smiley:

1. The separate instance for backup is intentional. We do open a 100% separate DuckDB instance to generate the backup, and yes, this allows DuckDB to temporarily use 2× the authorized limit during the backup. We do this to isolate the backup from production and especially to be able to release the RAM at the end: closing the instance returns all its memory to the OS. If we reused the production instance, we would certainly stay under ×2 during the backup, but the export would fill the production buffer pool to 100% of the limit without ever releasing it, for DuckDB it’s cache, it keeps it forever. Now keeping the entire database in cache because of a 10-minute backup per night is not great ^^

2. In your case, the real problem comes from LXC. The RAM limit of the container is not visible from the inside: Node calculates the 30% on the 64 GB of the server, i.e. ~19 GB of DuckDB limit… way too much for an 8 GB container, and ×2 during the backup. In the meantime, you can force an appropriate limit in your container’s config, for example:

DUCKDB_MEMORY_LIMIT=2GB

(~25% of the 8 GB allocated, leaving room for the backup peak.)

3. Why only now? You probably hadn’t seen the problem in the previous months because your database was smaller. With the accumulation of states, the export uses more and more RAM, until it triggers the OOM.

To improve the automatic detection of the memory limit, I have some leads (reading the cgroup limit via process.constrainedMemory() from Node), but as far as I remember we had done quite a few unsuccessful tests on LXC…

To re-test again, could you run these 3 commands and post the result?

What Node sees from the Gladys container:

docker exec gladys node -e "const os=require('os'); \
console.log('os.totalmem               =', Math.round(os.totalmem()/1048576), 'MB'); \
console.log('process.constrainedMemory =', Math.round((process.constrainedMemory()||0)/1048576), 'MB'); \
console.log('process.availableMemory   =', process.availableMemory ? Math.round(process.availableMemory()/1048576)+' MB' : 'n/a')"

The cgroup limits seen from the container:

docker exec gladys sh -c 'echo "cgroup v2 : $(cat /sys/fs/cgroup/memory.max 2> /dev/null || echo absent)"; \
echo "cgroup v1 : $(cat /sys/fs/cgroup/memory/memory.limit_in_bytes 2> /dev/null || echo absent)"; \
echo "meminfo   : $(head -1 /proc/meminfo)"'

The limit that Gladys actually applied:

docker logs gladys 2>&1 | grep -i "memory_limit"

If process.constrainedMemory returns your 8 GB (or if memory.max / cgroup v1 shows the real limit), we can automatically correct the calculation for everyone. If it returns 0 or the host’s RAM, we’ll rather go for more documentation on DUCKDB_MEMORY_LIMIT.