Hi,
I don’t know if I’m the only one, but I regularly get OOM errors when I allocate 8Gb to Gladys. This causes Gladys to restart every night.
After days of debugging without understanding the cause, I used Claude to move forward and identify a better hypothesis. And in the following days the problem was identified: the way the database is backed up.
Context: I keep all states indefinitely, as long as I have disk space.
Am I the only one this happens to?
Bug: the backup Gateway can cause an OOM-kill of the Node process
Components affected:
server/lib/gateway/gateway.backup.js
server/models/index.js (function duckDbCreateBackupInstance)
The problem: duckDbCreateBackupInstance() opens a new separate DuckDB instance on the same .duckdb file as the main instance (already active in read/write):
const backupDatabase = await DuckDBInstance.create(duckDbFilePath, {
memory_limit: duckDbMemoryLimit,
access_mode: 'READ_ONLY',
});
However, the official documentation of the Node.js « Neo » client for DuckDB explicitly states:
« Multiple instances in the same process should not attach the same database. »
The mechanism intended to avoid this is DuckDBInstanceCache.getOrCreateInstance(path), which reuses the same instance (and thus the same buffer pool / memory_limit) for the same file. The current code directly calls DuckDBInstance.create(), which creates two independent buffer pools, each capped at the same DUCKDB_MEMORY_LIMIT.
Result: the worst-case memory usage of DuckDB alone becomes 2 × memory_limit instead of 1 × memory_limit.
Worsening: the DuckDB documentation also specifies that memory_limit « only applies to the buffer manager » — compression buffers (GZIP compression used in EXPORT DATABASE) can consume in addition to this limit.
Proof in production (2 identical crashes, logs included):
- RSS stable ~2.5Gb before backup (including ~1.2Gb of DuckDB BASE_TABLE)
- Jump to ~4.6Gb in less than a minute, right after the log « Backing up DuckDB into a Parquet folder »
- Process killed (ExitCode=137) before reaching the log « Closing DuckDB backup instance to release memory » in the finally — so the crash occurs during the EXPORT DATABASE … FORMAT PARQUET COMPRESSION GZIP call itself, not after.
- Reproduced identically for 2 days in a row, always at the time of the daily Gateway backup.
What is requested:
- Make sure that duckDbCreateBackupInstance() uses DuckDBInstanceCache.getOrCreateInstance() instead of DuckDBInstance.create(), so that the backup shares the buffer pool of the main instance rather than opening a second one.
- Consider replacing COMPRESSION GZIP with ZSTD or SNAPPY in the EXPORT DATABASE (less compression buffers, faster export).
- Possibly document that DUCKDB_MEMORY_LIMIT should be sized anticipating that two instances may coexist during a backup (so recommend a value ≤ 25% of the container’s allocated RAM, not 30-50%), as long as point 1 is not fixed.
Context to situate the impact: installation with ~8Gb allocated to the container, BASE_TABLE of about 1.1-1.2Gb of sensor history — so not an extreme case, this behavior can probably affect other installations with a significant history.
