Gladys container crash / memory full at night

If you ever have an idea to automate this, because manually setting the limit is really a very clean solution, it’s still crazy that LXC doesn’t isolate that part…

If I find anything, I’ll let you know.

1 Like

It was me with my Gladys running under LXC.
I’ll also check if I can find a way to determine the exact RAM exposed by Proxmox.

1 Like

I searched a bit but each time it returns the total physical RAM and not the one exposed to my LXC.

I asked the AI and here’s what I got back

Gladys should use a smarter RAM calculation function when starting the DuckDB service. Here is the target algorithm:

1. **Check the Cgroup (The real limit):** Read `/sys/fs/cgroup/memory.max`.
  * If it's a number (e.g.: 8 GB): Use this value as the base.
  * If it's `max` or an error: Use `os.totalmem()` (32 GB) as the base.
2. **Apply the percentage (30%):** Calculate 30% of the chosen base.
3. **Apply a "Safety Cap" (The safeguard):** Even on a 128 GB server, DuckDB in Gladys doesn't need 40 GB of RAM to store temperature histories.
  * Cap the result at **2 GB** (a value more than sufficient for 99% of users).
* On a Raspberry Pi (2 GB):** DuckDB will take ~600 MB (30%). That's perfect.
* On a Proxmox LXC (8 GB on a 32 GB host):
    * If Gladys detects the 8 GB → it will take 2.4 GB, but will be limited by the cap to **2 GB**.
    * If Gladys fails to detect and sees 32 GB → it would want 9 GB, but will be throttled to **2 GB**.

And even suggests a sample code:

const fs = require('fs');
const os = require('os');

function getDuckDbMemoryLimit() {
  // 1. Check ENV variable
  if (process.env.DUCKDB_MEMORY_LIMIT) return process.env.DUCKDB_MEMORY_LIMIT;

  const SAFETY_CAP_GB = 2; // Safety cap for home automation
  let totalDetectedRam = os.totalmem();

  // 2. Try to read Cgroup limit (LXC/Docker)
  try {
    const cgroupLimit = fs.readFileSync('/sys/fs/cgroup/memory.max', 'utf8').trim();
    if (cgroupLimit !== 'max') {
      totalDetectedRam = parseInt(cgroupLimit, 10);
    }
  } catch (e) {
    // Fallback to os.totalmem()
  }

  // 3. Calculate 30% of detected RAM
  const calculatedLimit = totalDetectedRam * 0.30;
  
  // 4. Return the minimum between calculated limit and Safety Cap
  const safetyCapBytes = SAFETY_CAP_GB * 1024 * 1024 * 1024;
  return Math.min(calculatedLimit, safetyCapBytes);
}

The AI is wrong about both proposals, unfortunately :smiley:

As tested previously, /sys/fs/cgroup/memory.max does not contain the value of available RAM.

As for the idea of a safety cap, it’s a bit of an easy way out and I disagree with that analysis. A user who has 16 GB of RAM available for Gladys will benefit greatly from having more RAM for DuckDB if they have a lot of data and complex graphs, as in the case of energy monitoring.

Capping it at an arbitrary value would limit performance for those with systems that have plenty of RAM, which is a shame :slight_smile:

/ sys / fs / cgroup / memory.max returns max

In this case, for an LXC under Proxmox I don’t see a solution other than manually limiting it in the docker run or docker compose.
I tested a lot of things but each time it returned Proxmox’s RAM but not the LXC’s.

Too bad! I’ll update the documentation.

This whole thing is crazy; you’d think the isolation is done properly and that each container only sees what it’s supposed to see.

Yes, I thought so too.
However, as @prohand said, there is no such issue with a virtual machine on Proxmox.

1 Like

So since the AI says a lot of bullsh*t, I asked the AI :rofl:
With Perplexity, I get this :

mount | grep cgroup2
cat /sys/fs/cgroup/memory.current

and it works fine in my LXC which has 6GB of RAM :

root@gladys:~# mount | grep cgroup2
none on /sys/fs/cgroup type cgroup2 (rw,nosuid,nodev,noexec,relatime)
root@gladys:~# cat /sys/fs/cgroup/memory.current
6377263104
root@gladys:~# echo $(( $(cat /sys/fs/cgroup/memory.current) / 1024 / 1024 ))" MiB"
6077 MiB

and the other method (not 100% reliable apparently) would be :

root@gladys:~# cat /proc/meminfo
MemTotal:        6291456 kB

[quote=« mutmut, post:70, topic:9895 »]
and it works fine in my LXC that has

I just tried it and it’s not reliable, especially at startup.

The first time I tried it returned 8GB even though I have 16GB.

image
When I changed my RAM the other day I hadn’t restarted the LXC.

So I restarted it and now the value increases little by little

root@saga:~# docker exec -it gladys cat /sys/fs/cgroup/memory.current
1023455232
root@saga:~# docker exec -it gladys cat /sys/fs/cgroup/memory.current
1027911680
root@saga:~# docker exec -it gladys cat /sys/fs/cgroup/memory.current
1029660672
root@saga:~# docker exec -it gladys cat /sys/fs/cgroup/memory.current
1031991296
root@saga:~# docker exec -it gladys cat /sys/fs/cgroup/memory.current
1029668864
root@saga:~# docker exec -it gladys cat /sys/fs/cgroup/memory.current
1030901760
root@saga:~# docker exec -it gladys cat /sys/fs/cgroup/memory.current
1029951488
root@saga:~# docker exec -it gladys cat /sys/fs/cgroup/memory.current
1030709248
root@saga:~# docker exec -it gladys cat /sys/fs/cgroup/memory.current
1030946816
root@saga:~# docker exec -it gladys cat /sys/fs/cgroup/memory.current
1031671808
root@saga:~# docker exec -it gladys cat /sys/fs/cgroup/memory.current
1031319552
root@saga:~# docker exec -it gladys cat /sys/fs/cgroup/memory.current
1022386176
root@saga:~# docker exec -it gladys cat /sys/fs/cgroup/memory.current
1024593920
root@saga:~# docker exec -it gladys cat /sys/fs/cgroup/memory.current
1044897792
root@saga:~# docker exec -it gladys cat /sys/fs/cgroup/memory.current
1045057536
root@saga:~# docker exec -it gladys cat /sys/fs/cgroup/memory.current
1279713280
root@saga:~# docker exec -it gladys cat /sys/fs/cgroup/memory.current
1279754240

If we convert the value to a better format using @mutmut’s formula

root@saga:~# docker exec -it gladys echo $(( $(cat /sys/fs/cgroup/memory.current) / 1000 / 1000 / 1000 ))" Go"
1 Go

Some more tests regarding your request for @_Will_71.

If I run docker exec -it gladys cat /sys/fs/cgroup/memory.current in my LXC or directly cat /sys/fs/cgroup/memory.current inside the Gladys container, I see 4GB each time.
That seems to correspond to the 4GB I had allocated initially when creating my LXC, and following the memory leaks I had, I increased the RAM to 6GB.
But why I don’t see the current 6GB is a good question that I’ll investigate.

Regarding cat /proc/meminfo run inside the Gladys container, it returns the 16GB of my host so that’s not good.

Looking at the Proxmox docs for creating/editing RAM, Proxmox seems to be based on cgroup v1 :
image
And following @_Will_71’s remark, I also have a memory.current from Docker that increases and decreases very slowly directly inside the LXC which shows 4GB as well :thinking:
It feels like RAM is allocated dynamically to the LXC, weird.

EDIT :
this seems to be confirmed here :

Monitoring of consumption is also improved thanks to memory.current, which displays consumption in real time.

I think so — it’s handled dynamically, because every time it crashed due to RAM I increased the value in Proxmox and the LXC came back up without a reboot.

In the LXC config file (so directly on the host), you can clearly see that it’s cgroup2 and we have the correct amount of RAM allocated to the LXC:


I get the impression that Proxmox offers all of the host’s RAM to the containers and only provides what is requested in the config file, which should allow changing the RAM on the fly without problems, with a purely software limitation.

@_Will_71
I just noticed there’s a subdirectory .lxc in cgroup, there might be things to recover from there.
What do you get if you run:

docker exec -it gladys cat /sys/fs/cgroup/.lxc/memory.current

On my system I get a stable but small number: 634880

I don’t have a .lxc directory in cgroup

ok

And what value for :

docker exec -it gladys cat /sys/fs/cgroup/memory.peak

This one gives me my 6GB (or so)

This gives me 7.6GB, far from my 16GB

I’ve made good progress on the topic and it seems very complicated to easily obtain the real amount of RAM from the Gladys docker :frowning:
Indeed, it will always take the host’s amount even if the LXC reports the correct amount.

The only solution I’ve been able to test and validate so far is to specify the amount of RAM we want in the docker run or compose.
An example with docker-compose

services:
  gladys:
    image: gladysassistant/gladys:v4
...
    mem_limit: 6g  # amount of RAM we allocated to the LXC (here, 6GB)
    memswap_limit: 7g  # amount of RAM+swap (here, 1GB of swap)

Once restarted, I get these values that can be used afterwards:

root@gladys-test:~# docker inspect gladys | grep -i memory
            "Memory": 6442450944,
            "MemoryReservation": 0,
            "MemorySwap": 7516192768,
            "MemorySwappiness": null,
root@gladys-test:~# docker exec -it gladys cat /sys/fs/cgroup/memory.max
6442450944
root@gladys-test:~# docker exec -it gladys cat /sys/fs/cgroup/memory.swap.max
1073741824

For the record, via Prplexity:

This is a normal and known behavior with Docker in unprivileged LXC + nesting=1 on Proxmox: lxcfs exposes the correct LXC RAM (4 GB) at the container’s host level, but Docker (and its containers) see the host RAM (16 GB) because nesting propagates the cgroup namespace without limiting Docker’s /proc/meminfo.

  • lxcfs rewrites the /proc/meminfo of the LXC rootfree -m in LXC = 4 GB.
  • Docker with nesting creates its own child cgroups → its /proc/meminfo reads the parent cgroup (LXC limit) but often falls back to the host if not propagated.
  • Unprivileged + fuse-overlayfs (tteck script) exacerbates this: Docker ignores lxcfs LXC.
1 Like