Hello,
following the retrieval of values from an MQTT device (index/energy consumption) and its analysis here, I see that for this device I have 32,433 values that in fact do not change (if we put aside the erroneous value) :
Of course I have more or less as many values for each day from this Index sensor (Tempo HP Jour Rouge) that do not change for several months.
I can’t imagine the size that must take up in my DB which is already more than 3GB
For indexes that do not change value, couldn’t we have a start value (the first of the day) and an end value (the last of the day) if and only if the values are the same throughout the day?
More generally for indexes, can’t we record/keep only the values that are different from the previous one each day?
This kind of optimization is clearly not our responsibility: it’s exactly DuckDB’s job, and it does it very well
Conversely, intentionally deleting similar values on the Gladys side would cause a loss of information: the absence of variation is information in itself. We’ve already often discussed this on Gladys, and I was never in favor of that approach.
Now that we rely on DuckDB, which handles this case very efficiently, there’s really no argument left not to keep this data
ok, so that’s fine with me if it doesn’t save much more space than that.
However, even if DuckDB only stores the data once (which is identical), it still has to store the timestamp, which is different each time…
What bothers me most (but I’ll still sleep well, don’t worry ) is that I have more than 30,000 data points per day per index, even though some don’t change for months (the red ones, for example), and this has been going on for a year — that’s a lot of data
I agree.
In the particular case of my Tempo indexes, I know that I will never have a red day between 01/04 and 31/08, so I don’t need the value information to be stored. But that only concerns me, who use teleinfo2mqtt which sends a JSON message over MQTT where all the information is sent.
Conversely, intentionally deleting similar values on the Gladys side would cause a loss of information: the absence of variation is information in itself. We’ve already often discussed this on Gladys, and I was never in favor of that approach.
