As for me, nice volume reduction:
Great work, thanks
3 Likes
Awesome!! What a huge discount for everyone
2 Likes
Okay, my turn to share my migration stats:
- Almost 17 million states
- 2 hours of migration
- 33 hours to purge SQLite states
- Database before migration: 13.9 GB
- Database after migration: 89 MB (-99.4%)
- Gladys backup before: 3.35 GB
3 Likes
@pierre-gilles maybe this is an opportunity to propose a short blog/promo article for/with DuckDB, no?
1 Like
I guess that’s planned! 
Good evening, migration completed as well, here are the results :
Avant :
Nice job — I’m excited, my DB is over 10 GB!
For now, I’m unfortunately stuck. It might be related to the Raspberry Pi 3 or to some corrupted data in the DB.
This is the error I have that crashes the server, in case it rings a bell for anyone:
2024-08-29T01:20:36+0200 <info> device.migrateFromSQLiteToDuckDb.js:39 (DeviceManager.migrateFromSQLiteToDuckDb) DuckDB: Migrating data from SQLite
node: ../src/duckdb/src/include/duckdb/execution/aggregate_hashtable.hpp:47: void duckdb::aggr_ht_entry_t::SetPointer(duckdb::data_t* const&): Assertion `(reinterpret_cast<uint64_t>(pointer) & SALT_MASK) == 0' failed.
Meanwhile I’ll run a test on a faster server and use an older backup of my DB.
Indeed it looks like a piece of data in the database is causing a problem 
you’ll need someone more expert to answer you on that, I don’t really know much about it!
On my side the reduction is still as spectacular:
Amazing! Thanks for sharing 
It’s planned! For now I was doing a « French » launch only to existing users, to see if everything runs well for everyone, and to have statistics to share.
Next, I’ll start the « international » launch, and I’ll post on Twitter tagging DuckDB this time so they’ll re-share it. I’ll then see with them whether Gladys can be featured as a use case in their communications.
Finally, phase 3 is to poke Home Assistant a bit more directly on this point, because we now have a real competitive advantage over Gladys on the core.
I went to the HA forum to check, and DB size problems are a recurring issue; users have the same problems as us here, which makes sense since they currently use SQLite by default. Some users use InfluxDB, but that’s something extra to configure; the default remains SQLite (MariaDB/PostgreSQL configurable, but with the same disk space issues as SQLite).
Regarding migration to DuckDB on the HA side, they’re clearly not there yet. I saw that @McFlyPartages had created a thread on their forum, and it hasn’t really taken off for now. It’s the only mention of DuckDB on their entire forum…
In short, we’re way ahead, and I fully intend to take advantage of it 
Awesome 
Amazing!
Are you on a Raspberry Pi 3 with a 64-bit OS? If you’re on a 32-bit OS, I don’t think it will work. You must be on 64-bit.
It’s crazy
2 Likes
The English Twitter post:
Please RT/like
4 Likes
Yes, if it’s just for you and you want to save 99% of the build time, build only for the platforms you need (amd64 is a few minutes, for example)
The build that takes the longest is the one for arm/v7 — I don’t think you have any arm/v7 device… ^^
1 Like
Tu es sur un Raspberry Pi 3 avec un OS 64 bits ? Si tu es sur un OS 32 bits, ça ne marchera pas je pense. Il faut bien être en 64 bits.
Linux rpi3 6.6.31-v8+ #1766 SMP PREEMPT Fri May 24 12:14:10 BST 2024 aarch64 GNU/Linux
I shared the migration on Reddit Node.js! Any upvote is welcome 
https://www.reddit.com/r/node/comments/1f3z6jh/migrated_our_nodejs_app_from_sqlite_to_duckdb_and/
Ok that looks good. It might be a dodgy value in your database, could you inspect your SQLite DB a bit? See if there aren’t any values in there that aren’t doubles?
It could be that a « string » slipped into your table, hence the cast error
2 Likes
Ah great thanks, yes indeed it’s Gladys for the professional part with my modifications to remove access to a lot of pages if not admin!!
Awesome!! It’s done!!
1 Like
Hello,
I’m on my way back from vacation and I see on my phone that the migration is stuck.
I haven’t changed anything on the system and the latest backups went from 11% to 17%. The update to the latest version correctly shows 4.45
1 Like
Congratulations on this release 
.
SQLite database: 37 GB
DuckDB database: 272 MB
Savings of over 99%
The SQLite purge was long (more than 12 hours).
Well done again.
2 Likes
Awesome
That’s crazy! You’re welcome 