Help! Problem with aggregation?

Do you mean a feature to delete old sensor values per sensor?

Yes exactly, a feature to allow each user to decide whether they want to delete the entire history at certain dates, keep or not the aggregated values at those dates, etc.

My DB is already 2.8GB, it’s starting to take up space :wink:

A solution for you already, you can set a maximum retention period in the settings:

If you set « 3 months Â», it will already remove everything older (every 4 hours, Gladys cleans the old values)

After all, a way to purge by device could be interesting

Would you create a feature request? :slight_smile:

For now, as we cannot natively consult the history, I will do this, but does it also delete the aggregation?
Because I would like to be able to keep the display for one year in the dashboards.

It works! However, I think Gladys is having trouble responding during aggregation, which took 21 minutes for daily values.

Since the aggregation is not calculated due to the bug, it will all be removed (since the aggregation was not done, you only have the raw data)

In your case, I think it’s worth going to the DB to delete the too verbose deviceFeatureState.

You can run this query to see the most verbose features:

SELECT COUNT(id), device_feature_id
FROM t_device_feature_state
GROUP BY device_feature_id
ORDER BY COUNT(id) DESC;

For info, I ran this query on the DB you sent me, and you have several features with states in the millions.

Then you do:

DELETE FROM t_device_feature_state WHERE device_feature_id = '';

Replacing with the ids you want to remove.

Since this is your first integration, if your Pi does not have many cores (or a slow disk), it is possible that it slows down Gladys during the first aggregation.

Normally, it’s not blocking (it’s a separate process), but of course the first aggregation takes a lot of resources :slight_smile:

Indeed, that’s possible. But why do my graphs still work for all periods in this case?

Why wouldn’t they work? I don’t follow :smiley:

Your aggregation seems to have worked, so you have access to the values on the charts.

You’re talking about « first aggregation Â», so without aggregated data how are the charts displayed?
If I understand correctly, according to you my data was never aggregated (or not completely), which for me should translate to « no chart Â».

Anyway, it’s resolved now! Aggregation is working on the monthly and daily! Thanks a lot for your investigation.

Exactly, normally you shouldn’t have had any charts without aggregation, except for the « last hour Â» chart which uses live data

This is indeed what I noticed in the database you sent me, without aggregation, the charts did not work!

Well, you’re mistaken, I had graphs for all periods: latest data, 24h, 7d, 30d :slight_smile:

So the aggregation had worked partially :slight_smile: Without aggregation, these views don’t work anyway

How to retrieve the names of the devices?
I retrieved a list of device_feature_id but I would like to remove only those for which I have little interest. I imagine that my connected plug sends a lot of values, which will not be the case for temperature sensors that are on battery.

I tried this randomly, but of course it doesn’t work:

sqlite> select * from t_device where device_feature_id = '12598df4-d2d6-4ded-98f6-bda3d7e3aa84';

I had suggested it here :wink:

@lmilcent Here you go:

SELECT COUNT(t_device_feature_state.id), device_feature_id,
t_device_feature.name as feature_name, t_device.name as device_name
FROM t_device_feature_state
JOIN t_device_feature ON t_device_feature.id = device_feature_id
JOIN t_device ON t_device.id = t_device_feature.device_id
GROUP BY device_feature_id
ORDER BY COUNT(t_device_feature_state.id) DESC;

For follow-up, @AlexTrovato proposed a PR that fixes the « NaN Â» issue in the DB that was causing the aggregation to crash:

:heart_eyes:

I actually have the complete request, thanks!!

The result is interesting, the sensor I use the least pollutes the DB the most. And I think the Ikea bulb generated a lot of records because of my tests with Node-Red (infinite loop for a while :sweat_smile::sweat_smile:).

1410655|12598df4-d2d6-4ded-98f6-bda3d7e3aa84|Temperature|AirQualitySensor
1410655|3bbf3246-2124-4d79-91f7-48f640739912|Humidity (percentage)|AirQualitySensor
1410655|6a667e61-b2a1-4114-86ec-d8ece88cd4a5|CO2 Level|AirQualitySensor
131520|f93f5258-e274-4f7d-bc43-21dd0af47b1f|Lighting Brightness|IkeaOfficeBulb
131519|8e8299a0-ace8-4867-b143-6755f3e47cc9|Lighting On/Off|IkeaOfficeBulb
118949|10c7fd52-52d8-4033-9d59-a5ce1cd2497d|Power Consumed|FridgeOutlet
118949|18123840-c7d8-41d8-ab0d-3443d928bac0|Current Consumed|FridgeOutlet
118949|1ccaa66b-6fbe-40d2-a6a8-b27206682f09|Switch|FridgeOutlet
118949|1fb46a04-52a4-488f-8fac-65a6bb1c55d9|Energy Consumed|FridgeOutlet
118949|3d7b2475-3ecd-488f-901a-000c4be2b8cb|Average Voltage|FridgeOutlet
117591|20243225-bfb8-4aa9-abaa-7057e7e0e49b|Average Voltage|OfficeOutlet
117591|6eb6a796-25fd-47a1-81a3-b955fde621ca|Energy Consumed|OfficeOutlet
117591|c64324f1-05d2-4988-971b-90353ee573d7|Switch|OfficeOutlet
117591|d59b5e7e-27eb-4599-bdc3-46788dc412d5|Power Consumed|OfficeOutlet
117591|e23c8dde-4169-45cd-a3ef-d7122b0f28cc|Current Consumed|OfficeOutlet
28535|54025171-7528-40cd-b09c-43eaadf7b97c|Battery|FridgeTempSensor
28535|5780926c-22a6-465f-bc68-f10303814611|Humidity (percentage)|FridgeTempSensor
28535|5851cce7-5dc0-4cc0-add3-03073e0143fa|Temperature|FridgeTempSensor
28535|dab823e7-c4e5-455f-96dd-1543f9f86ef8|Decimal Pressure Value|FridgeTempSensor
24035|0644e4d0-c50f-48ac-8234-610ae320a79d|Temperature|EntranceMotionSensor
24035|2b204d50-fd98-4ff6-bfd5-fa591496bdb7|Decimal Brightness|EntranceMotionSensor
24035|7c36d2c8-ed15-4774-a9ad-981f7bd791ef|Average Voltage|EntranceMotionSensor
24035|ac2d6499-8f50-4a73-a864-dcd70681e139|Motion Detection Yes/No|EntranceMotionSensor
24035|cbe2ab38-47e2-4b2a-be55-475330ac9efe|Battery|EntranceMotionSensor
19979|84728634-408c-4c07-89af-eb345cfc5b5e|Humidity (percentage)|FreezerTempSensor
19979|932922bb-692d-4790-a965-a28f9c392eb6|Decimal Pressure Value|FreezerTempSensor
19979|c89b14fb-dc72-418c-b426-220ec23d3509|Battery|FreezerTempSensor
19979|f1df07da-6602-4332-839e-bf4836e9b22d|Temperature|FreezerTempSensor
19973|82187e9e-201d-44c0-b820-23662f3bc6c7|Humidity (percentage)|DressingRoomTempSensor
19973|df809a38-e1c0-4fb1-a546-105c6b6dc4de|Battery|DressingRoomTempSensor
19973|e2008643-fd03-4900-b113-4ea23bb8bb1d|Decimal Pressure Value|DressingRoomTempSensor
19973|fae472e2-979d-4e70-96a5-661c4dc65ae1|Temperature|DressingRoomTempSensor
18037|1d59e9db-c093-4e11-804b-bcff945062f2|Motion Detection Yes/No|LivingRoomMotionSensor
18037|65cda733-9e4b-48b0-b33b-67c4301432ed|Decimal Brightness|LivingRoomMotionSensor
18037|6c8c8245-7b5c-4f9e-a314-fdc12bb70fd8|Temperature|LivingRoomMotionSensor
18037|b2e6b893-acbb-4024-a766-3c75fac54a36|Battery|LivingRoomMotionSensor
18037|db74e9f6-a92e-4e0f-82b4-f1cbb6c1c683|Average Voltage|LivingRoomMotionSensor
17505|7e4dda24-ed81-42ce-9aa8-4f2129e9bcce|Motion Detection Yes/No|DiningRoomMotionSensor
17505|a1da4359-784c-49f2-8198-5796d2ad90b7|Temperature|DiningRoomMotionSensor
17505|a3eb52bd-5081-47b2-8c09-eac4c8ff337b|Average Voltage|DiningRoomMotionSensor
17505|a9d944ec-1f4c-4a02-9ffa-ce149410946f|Decimal Brightness|DiningRoomMotionSensor
17505|c11cf1dc-5eef-4beb-a956-90eb60def17f|Battery|DiningRoomMotionSensor
15106|a2eb5b0f-e5d7-4349-a6d9-64c12ec04bae|Temperature|OutdoorTempSensor
15105|48071ec8-c62d-4696-b281-0f384b22d1b1|Humidity (percentage)|OutdoorTempSensor
15104|214ba796-d0d8-43b7-9a66-50533933ea8f|Decimal Pressure Value|OutdoorTempSensor
15040|09d2e716-bee6-4adf-b0e7-a8f53a053321|Battery|OutdoorTempSensor
13579|5732c162-c5f6-4e56-856b-ae723093456a|Temperature|BathroomTempSensor
13579|b2407be1-b09c-479d-92a7-c989bdf59060|Humidity (percentage)|BathroomTempSensor
13579|bbe97fb5-25b8-4c74-93b5-23d1e8838709|Decimal Pressure Value|BathroomTempSensor
13579|d4b2f302-803f-4507-a572-fcc70ada1a0f|Battery|BathroomTempSensor
10595|2f82f2da-8a49-47fc-9737-75e18739a0db|Humidity (percentage)|DiningRoomTempSensor
10595|5377cb83-a984-4aac-8937-1646fc8510a9|Decimal Pressure Value|DiningRoomTempSensor
10595|6476a45c-8361-4233-92e6-1e14da76c48b|Temperature|DiningRoomTempSensor
10595|af7eb901-dc34-4e5c-8579-55c0428f5ae7|Battery|DiningRoomTempSensor
10415|0c81f820-b44d-497b-acff-12e64b9c6204|Temperature|ParentsBedroomTempSensor
10415|25ffdab9-2d5c-4635-bb4c-b2b50cef3d99|Humidity (percentage)|ParentsBedroomTempSensor
10415|28402eba-ab8f-4fb4-8e20-737af9907ba3|Battery|ParentsBedroomTempSensor
10415|e6fa4083-fcb0-4793-bf59-5f5906777416|Decimal Pressure Value|ParentsBedroomTempSensor
10163|0ac988ae-3b72-4407-95d0-158f25ff2935|Humidity (percentage)|ChildrenBedroomTempSensor
10163|5473d6e9-39a3-4eaf-9690-43d0e3804a4e|Temperature|ChildrenBedroomTempSensor
10163|54d6dc9c-16ea-4e2b-ac6b-390c4642f6d3|Battery|ChildrenBedroomTempSensor
10163|f803a4b6-c32e-4b5b-94c7-4b6d32a0279b|Decimal Pressure Value|ChildrenBedroomTempSensor
8964|19077f2e-782c-4977-b15a-9d0b7f0e293c|Temperature|CellarTempSensor
8964|e0425041-f05b-4465-b87f-762ae6eb0b8b|Decimal Pressure Value|CellarTempSensor
8964|fee03c4c-a320-4a10-b80d-4c8877edc464|Humidity (percentage)|CellarTempSensor
8877|0a62d20e-f68a-4137-b077-177ecb1b87f4|Battery|CellarTempSensor
8877|5813c2ba-0ef9-4594-bdff-e9d10a4d57c2|Average Voltage|CellarTempSensor
8297|0704435d-6980-49c1-b7a1-ba4a8a70d8ef|Temperature|KitchenTempSensor
8297|2e188de0-1c03-4f7a-b8e7-04a186e54fbb|Decimal Pressure Value|KitchenTempSensor
8297|ba25a3f7-be08-4595-a02b-6bc1ba41277b|Battery|KitchenTempSensor
8297|cb332692-7844-4514-a2fd-ef3c90ed0121|Humidity (percentage)|KitchenTempSensor
7396|06c18a65-9695-494d-9384-051268e74b5f|Humidity (percentage)|LivingRoomTempSensor
7396|0cdc0014-d990-420c-8cb9-efccd2ba72c4|Temperature|LivingRoomTempSensor
7396|3af4d686-d454-4137-a822-b403f612c4c9|Battery|LivingRoomTempSensor
7396|3fb1459f-44ad-4d9e-ae5f-b7c5da0f9cb9|Decimal Pressure Value|LivingRoomTempSensor
3744|0416dcbb-004b-442e-9632-eb482e9c5688|Switch|LivingRoomRadiator
3744|24d37ef5-8f09-4ac6-840b-c72b9b21a54f|Temperature|LivingRoomRadiator
3744|b6175530-9d75-44dc-b6e3-60ac68bd87e2|Battery|LivingRoomRadiator
3744|e5da6401-e688-4b6b-9312-ccc6cc5a3e2e|Switch|LivingRoomRadiator
3472|25a4da86-8072-4072-a7e0-4c4d5a7c7505|Battery|KitchenOpeningSensor
3472|44f77ee7-5faa-47ad-82f7-20e05d5b1d5b|Opening (Yes/No)|KitchenOpeningSensor
3472|461c5461-3a6e-4ba0-81e4-93133523bfba|Temperature|KitchenOpeningSensor
3443|a3946d88-9a4b-4352-9077-e8fb2d6756d8|Battery|WirelessIkeaSwitch
3297|2534501d-5e7c-4f3e-91e5-d4858115382b|Temperature|EntranceOpeningSensor
3297|29404efd-f8f6-446d-bfe8-8d1370f8ee43|Opening (Yes/No)|EntranceOpeningSensor
3297|edd22742-aa61-4f3c-8e3c-0d34e22d6f87|Battery|EntranceOpeningSensor
3239|e011e9d1-9ce8-4851-bebe-8a977087a76f|Average Voltage|FridgeTempSensor
2962|f8e9ce4e-40d2-483f-94fe-b7c8335f1cc7|Average Voltage|FreezerTempSensor
2613|2f45b03d-135e-489f-93c2-9028a682b7a1|Battery|DiningRoomRadiator
2613|9516bc96-df01-4bea-8e6d-6226d564589e|Switch|DiningRoomRadiator
2613|b59cf77d-d941-458f-a38f-631e75beea3f|Temperature|DiningRoomRadiator
2613|f4ee7cca-ccaa-431d-b532-92f39ec5ce60|Switch|DiningRoomRadiator
2156|3ccacfa6-66e6-4205-8e6a-d14a54842d90|Opening (Yes/No)|DiningRoomOpeningSensor
2156|9433ed15-3628-4d1c-a5eb-423df28665a7|Temperature|DiningRoomOpeningSensor
2156|adbf00cc-f134-4dcb-82d9-c730d9eb7865|Battery|DiningRoomOpeningSensor
2123|545d6837-65f1-4d24-90d0-be069fbdb761|Temperature|ChildrenBedroomOpeningSensor
2123|ea28bdd2-d6fc-4f70-a343-b2e38f74eb9d|Battery|ChildrenBedroomOpeningSensor
2123|ee14ad4b-507c-4939-88c4-3bfc5dfe7183|Opening (Yes/No)|ChildrenBedroomOpeningSensor
2090|18b92445-8eaf-4867-9607-eddd4ed23624|Temperature|LivingRoomOpeningSensor
2090|934642c3-0c81-499d-88ec-178c25c0d7c0|Battery|LivingRoomOpeningSensor
2090|b3748079-47b1-48b1-ac64-dba9e55becb6|Opening (Yes/No)|LivingRoomOpeningSensor
2053|6fb12b59-378e-4510-860d-b68604124561|Temperature|KitchenWindowOpeningSensor
2053|dec844b4-f6f9-422b-bf39-5e44af08e5e8|Opening (Yes/No)|KitchenWindowSensor
2053|e72c4802-1d26-478c-9c8d-83adc6280ce5|Battery|KitchenWindowSensor
2036|06bd5552-4d6f-40f2-b91c-565143358b29|Opening (Yes/No)|ParentsBedroomWindowSensor
2036|2d39ac0b-8da8-4871-945a-bb479ab51f75|Temperature|ParentsBedroomWindowSensor
2036|56ba42fa-65e0-42d9-9610-1ae1273e0be1|Battery|ParentsBedroomWindowSensor
2001|6ca36652-d2ab-4097-b192-ee8e4aa5a908|Battery|WirelessSwitch
1946|0fb6d3af-c01c-4a24-a4b9-a7496cc62947|Battery|LivingRoomGWindowSensor
1946|3e4c51db-0592-4950-b9e0-f14b7033ab66|Opening (Yes/No)|LivingRoomGWindowSensor
1946|99884515-5aed-4ad1-bc40-79b3ca76de0e|Temperature|LivingRoomGWindowSensor
1941|1269ab99-0364-44b3-96af-fa451f79b340|Temperature|BathroomWindowSensor
1941|9c5e580a-f4ef-4465-ba6d-996998998764|Opening (Yes/No)|BathroomWindowSensor
1941|f80f794a-5a5a-4723-9296-a7f5e560fd68|Battery|BathroomWindowSensor
1938|38ab06f3-743d-40c8-be69-b40d46b5f42e|Average Voltage|MailboxMotionSensor
1938|3a0ea935-cd7c-4942-aa04-a98311ae430f|Battery|MailboxMotionSensor
1938|7f537027-c0aa-4963-a9ef-9abbbb63b000|Decimal Brightness|MailboxMotionSensor
1938|86629a3e-8df7-40e1-b222-5e7abb3cdce8|Temperature|MailboxMotionSensor
1938|fbb2b9c2-ce1e-4204-a8a4-813732d7a526|Motion Detection Yes/No|MailboxMotionSensor
1921|3e4c6d31-82f8-4cd1-bcd1-ae2306f94a5b|Opening (Yes/No)|DressingRoomWindowSensor
1921|623911ff-3602-4acd-bab8-e142d6e493d0|Temperature|DressingRoomWindowSensor
1614|48dd81f3-e5f2-43fe-969f-fa76b1f6aec7|Battery|OnePlus 7T Pro
1596|b9d9578d-5b72-4ded-ae0b-1c0be5209a6b|Average Voltage|OutdoorTempSensor
1571|c74cf1a0-10b4-43b0-adb0-28a19ded64d8|Average Voltage|EntranceWindowSensor
1506|4ef54862-74ca-4b36-a79d-108ec8eba050|Switch|KitchenSwitch
1299|fc4f0e26-3445-42ec-a56a-b68d86147521|Average Voltage|BathroomTempSensor
1258|b5345b18-b96d-409f-908b-c088597a0dde|OP7 Presence|OnePlus 7T Pro
1002|0d72a315-58ac-4b48-a7fc-a82e38283629|Average Voltage|LivingRoomTempSensor
1001|4408364a-f235-40ee-9f91-41b8e6ddd710|Average Voltage|DiningRoomTempSensor
988|2841ad26-b82a-4eab-8ccf-56b77f853f44|Average Voltage|DiningRoomWindowSensor
988|79827b2c-b8e4-4d59-9a8f-d2266f39f763|Average Voltage|ChildrenBedroomTempSensor
987|47b0e3e0-4431-486b-8065-0f79783d37d1|Average Voltage|ParentsBedroomTempSensor
797|031660bf-26e2-4bbe-836d-433d93ff4ee5|Average Voltage|KitchenTempSensor
596|96e636ba-d653-40d0-8b81-72577d8a50f0|Switch|ChildrenBedroomSwitch
541|1edbc42b-bbdb-45b0-826d-1db635043dd3|Battery|IkeaMotionSensor
541|41480aad-bfba-4baa-b433-0cc470ff8608|Motion Detection Yes/No|IkeaMotionSensor
333|b0190589-478e-4a2a-aa7a-558d3ac6f30f|Average Voltage|KitchenWindowSensor
296|fa53140a-c624-46b0-9348-e5dd005c0f39|Walk Bug - Step 1 |Walk Bug - Step 1
263|4bb63ee9-7c05-4a04-8c52-cba1ceef7d64|Switch|LivingRoomSwitch
218|3a803961-21b9-4f06-91c0-91141614e5e7|Switch|ChristmasTreePlug
212|2d3740ca-3e93-4ab0-a7b3-7c0ac7064bf5|Switch|GarlandPlug
191|6742e9d7-3e27-415b-a662-9298513b166b|Average Voltage|WirelessSwitch
182|9d4cf46a-d559-4876-8598-8f78b575a73e|Average Voltage|ChildrenBedroomWindowSensor
178|05c11039-5ec8-4297-bdf2-0562e62d95be|Average Voltage|KitchenWindowSensor
169|2dbbc49b-6a3b-484e-ae63-67b492617bc3|Average Voltage|ParentsBedroomWindowSensor
169|f8f8c4fc-ca19-4a1f-b5dc-68d93454e0cf|Average Voltage|LivingRoomGWindowSensor
166|daef3db3-1227-43a7-871d-0cd4e60c879f|Battery|DressingRoomWindowSensor
166|ddfd1688-fb2c-4275-8599-cf20b59b03f0|Average Voltage|LivingRoomDWindowSensor
166|f7c0d98f-f3ca-4475-823d-c46ddc9674b1|Average Voltage|DressingRoomWindowSensor
163|4f5cf0dd-3f10-440b-888e-f2039cf53762|Average Voltage|BathroomWindowSensor
122|3805996c-e611-4fd9-91b0-c7098e8f8eb9|22h - 1h Period Value|22h-1h Period
116|fc350de6-6802-4d41-9176-fb0383a718ef|Button Click|WirelessSwitch
14|6d6e13f8-8055-4630-a23c-a6685578714d|Access Control Mode|LivingRoomRadiator
5|68ae567e-2be2-4472-bbaa-135dca94c7e9|Average Voltage|DressingRoomTempSensor
5|7786f87c-789d-4ce3-ae1f-617cab63eeb4|Access Control Mode|DiningRoomRadiator

After removing the first lines related to "AirQualitySensor", "IkeaOfficeBulb", here are the DB sizes:
* Before: 2.8GB
* After: 2.8GB đŸ€”

Not sure you’ll notice a significant change until the db is closed. To be confirmed

When a line is deleted, most databases (including SQLite) do not directly delete it from the physical file, often for performance reasons. The database is often cleaned periodically or at startup/shutdown.

If you want SQLite to rebuild your DB file, you can run the SQL command in your DB:

VACUUM;

This will clean your DB and only keep the non-deleted data :slight_smile:

See: VACUUM