More granular management of state history

In Gladys, currently it is only possible to select:

The « aggregated » states are currently retained indefinitely.

The idea of this development would be to offer more fine-grained management of this area:

And to allow choosing a different retention period for aggregated states:

What do you think? Do the durations look good to you or should I add more?

I think that’s very good, it provides quite a few options; I admit I can’t really think of any particular case that would require even more duration options.

@guim31 what duration do you use?

3 months, without really ever having needed to go back any further.
Because the only thing I might want to consult/compare from one year to the next would be my electricity consumption, but I do that from the dedicated app for my solar panels which is very comprehensive.

Okay, but right now you’re keeping all the aggregated data indefinitely even though you only want it for 3 months ^^ So that’s not optimal :smiley:

Hence this project

2 Likes

I’m voting straight away because the database keeps growing, so I clean it myself from time to time to delete aggregated data.

1 Like

Yes, exactly!

The pull request:

2 Likes

It’s true that more granular management will also limit the size of backups, because having only 3 months and unlimited was too restrictive — it’s good that these durations were added! :+1:

I took the opportunity to add this task to the « background tasks » screen:

2 Likes

So it’s retroactive?
If I first set it to unlimited, and then 1 year later set it to 1 month, will all the previous states be deleted as a result?

Yes, that same night at 4 a.m. it will be purged.

1 Like

""in the evening" and "4 AM", that’s not supposed to make sense :joy:

Thanks for the confirmation that, for now, aggregated data are kept indefinitely — I had forgotten.

A more or less silly question: I think you’ve already answered this, but topics sometimes evolve:

2 Likes
1 Like

On that, I’ve been eyeing for a while a new file-based DB (like SQLite) but for time-series data; it’s called DuckDB and it’s becoming somewhat of a reference. I tested it and the space it uses compared to Gladys right now is pretty phenomenal (for time-series data, of course)

I was waiting for DuckDB to go stable to start work on it

As for InfluxDB for those who know it, why not have an integration, but for me it wouldn’t be a « recommended » way to do it but rather an integration for experts :slight_smile:

1 Like

Indeed, you had already mentioned this project. I can’t wait for it to enter stable mode, because we’ll all be able to benefit from it:

  • Fewer SSD/MicroSD accesses on Raspberry Pi, I think
  • Ability to keep all our sensor data for more than a year, without disk space issues
  • Optimized backups on the Gladys Plus side (my DB is over 10 GB!!)

I think the same, maybe even more, don’t get your hopes up about that aha :smiley:

But that, yes definitely — that’s the whole point.

1 Like

@lmilcent Here, if you want some hope, I ran a test with a 4.5GB DB of sensor states (varied data) imported into DuckDB.

  • In Gladys: 4.5GB
  • In DuckDB: 50Mb

We’re looking at a factor of 90, it’s pretty crazy :stuck_out_tongue:

(Well, it’s normal, it’s a time-series DB, it’s made for that)

3 Likes

Even knowing it’s designed for it, it’s impressive!

Currently 14 GB of database would yield 160 MB with a factor of 90. :star_struck:

1 Like

14GB is on disk, right? On the Gladys Plus side, compressed + encrypted, how big is it? Are your backups still working?