For info, I did quite a few tests with ChatGPT. I notably restarted the gladys prod container with the following option to create snapshots.
So with Gladys stopped, I’m at 4.03 GB of memory in use.
Once Gladys finished starting, I’m at 5.7 GB of memory (15:43).
Until 16:00 I stayed in that range, with only a few fluctuations. But you can see the memory is being freed properly.
At 16:00 the consumption / cost calculation did occur with a slight memory increase but was immediately freed. That confirms it’s not that. Tasks completed.
1 minute later, I go to the dashboard (I couldn’t say if that’s the trigger) and poof I instantly jumped to 8.82 GB: Column « RES » (= reserved memory) goes from 1748MB to 4995MB. => Nothing is freed
I’ll paste raw what ChatGPT tells me, and we can see after if you want anything else (Conversation: https://chatgpt.com/share/693d82d6-93e8-800b-8686-ad0b41963f18)
Yes. Here we clearly see what’s happening, and you were right from the start: this is not a JS heap leak, it’s a native memory explosion (outside V8), triggered by a one-off operation.
I’ll give you a precise, factual readout based on the “peak” report you just provided.
What the report says at the time of the peak (numeric evidence)
JS Heap (V8)
"usedMemory": 252011200 ≈ 240 Mo
"externalMemory": 25243016 ≈ 24 Mo
Heap perfectly stable, almost identical to previous snapshots.
No JS leak.
Total process memory
"rss": 6198140928 ≈ 5,77 Go
"maxRss": 6374838272 ≈ 5,93 Go
+4 GB outside the JS heap that appeared abruptly.
Linux side (/proc)
RssAnon: 5 089 920 kB ≈ 4,85 Go
RssFile: 59 392 kB
Private_Dirty: 5 090 240 kB
Private anonymous memory
Not file-mapped
Not freed afterwards
Typical signature:
massive native allocations (malloc / new on the C/C++ side)
What it is NOT (important)
Not:
- V8
- Garbage Collector
- Retained JS objects
- Timers or blocked event loop
- Classic Node leak
Your snapshots prove it:
the heap stays at ~260 MB while RSS explodes to ~6 GB.
What it is (almost certain at this stage)
Let’s look at the loaded native bindings:
"/src/server/node_modules/duckdb/lib/binding/duckdb.node"
"/src/server/node_modules/sqlite3/lib/binding/node_sqlite3.node"
"/src/server/services/usb/.../bindings-cpp.node"
"/src/server/services/bluetooth/.../bluetooth_hci_socket.node"
Primary suspect #1: DuckDB
This is THE main suspect, for several reasons:
- DuckDB allocates heavily in native memory
- Uses:
- vectorized execution
- columnar buffers
- intermediate caches
- Doesn’t always return memory to the OS immediately
- Can grow by several GB on a single query
And especially:
The peak is instantaneous, high CPU, memory never goes back down
→ exactly the observed behavior.
Why it doesn’t go back down
Very important to understand:
- DuckDB logically frees its buffers
- BUT:
malloc() keeps memory in the arena- the RSS does not go down
- Node will reuse this memory later, but Linux still sees it as « taken »
So:
This is not an infinite leak
It’s a stepwise irreversible increase
The likely trigger in your setup
Given your Gladys + Energy Monitoring + history setup:
- energy aggregation
- temporal calculations
- analytical queries
- possibly:
- daily consolidation
- aggregates per phase / device
- historical recalculation
A heavy DuckDB query was launched (cron, internal job, UI action).
This matches:
- CPU working
- RAM rising in 1 second
- no visible JS allocations
Absolute confirmation (if you want to be 100% sure)
During the next peak, run immediately:
docker exec gladys bash -lc '
cat /proc/1/smaps | awk "
/duckdb.node/ {f=1}
^$/ {f=0}
f && /Rss:/ {print}
"
'
If you see hundreds of MB / GB associated with DuckDB → final verdict.
What to do concretely (realistic solutions)
Option A – Limit DuckDB (recommended)
In Gladys (code):
- limit query sizes
- avoid global aggregates on the full history
- chunk by period (day / week)
Option B – Force a Node memory cap
This does not fix the root cause, but prevents host OOM:
-e NODE_OPTIONS="--max-old-space-size=2048 ..."
Note: this does not limit native memory, but avoids stacking the two.
Option C – Scheduled restart (pragmatic)
If the peaks are rare but destructive:
docker restart gladys
- weekly
- or after a heavy task
This is very common for DuckDB in production.
Clear and concise conclusion
Your data is excellent
The diagnosis is now solid
There is no JS memory leak
RAM is consumed by native allocations (very likely DuckDB)
Memory doesn’t go back down because it remains in the native arena
If you want, next possible steps:
- identify which Gladys query triggers the peak
- or help you patch / work around on the Gladys side (SQL logging, throttle, split)
See you later!
We need to find which part of Gladys is responsible.
