More granular management of state history

Yes, we agree :wink:
3.25 GB of backup!

It’s true that if we keep all data logging enabled it becomes substantial; I currently have about ten sensors and they already take up 350 MB. After that, how are the data saved (on every value change, or according to a timer)? Is it necessary to save everything or only based on thresholds (the battery for example: it’s only if it drops below a threshold that it may be useful to know the duration, or else by steps 90% 85% etc.) There are variations in usage (temperature, humidity, etc.) and variations in maintenance (battery, voltage, current, etc.) — the two don’t cover the same needs so data storage could be different!

That’s crazy, these weight differences — how is that possible?!

It would take a long time to explain, but the takeaway is that on the Gladys side, to achieve fairly fast querying of time-series data, we duplicate the data several times (the so-called aggregated data), and we use indexes that are themselves duplicated data. Put together, that creates a lot of duplicates :slight_smile:

In a time-series database, the data format is different and optimized for temporal data, so there’s no need to do the tricks we do with a relational DB. On top of that, a time-series DB uses compression algorithms to store the data compressed and can decompress on the fly when reading.

That said, all that has trade-offs: these time-series databases are only good for temporal data, so if one day we implement this in Gladys, it will be only the « t_device_feature_state » table that moves to that DB — the rest will remain in SQLite (which is ultra-robust for anything relational)

3 Likes

This feature is available in Gladys Assistant 4.31 ! :rocket:

I’m closing this thread to free up the votes, feel free to create another thread if there are bugs with this feature :slight_smile: