The road to Gladys 4 RC

News
1

Hello everyone!

After a few weeks of vacation, I’m back on Gladys :rocket:

The goal is now clear: to take Gladys v4 out of beta and move to a stable version.

Remaining tasks

Here is my remaining to-do list before releasing Gladys v4:

My goals

Some statistics on the beta, there are currently 104 Gladys instances running in a « long-term » manner, in production (which have been active for several weeks, these are not test instances)

Screenshot 2020-10-12 at 15.08.20
Screenshot 2020-10-12 at 15.08.201700×468 29 KB

An interesting figure I follow is the « retention » of Gladys installations. Do users who install Gladys keep it for a long time, or uninstall it quickly?

So far, we have an excellent retention rate of 80%. This means that 80% of people who install Gladys keep it active afterwards. This figure is likely to drop when we exit beta, as the number of installations will grow, but the goal is to keep it as high as possible.

The goal, after the release of this v4, is to reach 1,000 Gladys 4 instances as quickly as possible. I think it is possible to achieve this goal within 6 months of the v4 release.

To do this, we will need to: communicate, communicate, communicate. We need to hear about Gladys everywhere in the home automation world.

If you know any journalists/bloggers in the home automation/connected object/tech world, don’t hesitate to give me their contact info :slight_smile: I would be delighted to talk to them about Gladys.

2

Hello everyone,

This morning I worked on merging the Z-Wave PR, and while the tests were running, I worked on the press release for the launch of Gladys 4.

For those who don’t know, the purpose of a press release is to send the media a sample article, written as if we were journalists, talking about our own product/release. All brands do this when they want to announce something.

I will send it to French home automation news sites/blogs with a press kit (photos of Gladys 4, logos), and this will allow these sites to write an article about the launch if they wish.

Here is the link to this document, I am working on it live but I am already open to your feedback if you have any ideas/suggestions.

Note: For now, it’s a jumble of ideas that will become an article written with nice sentences. It’s a work in progress!

3

Hello

I saw a beta 16 pass by, I suppose it’s the one that integrates the zwave fixes?

Because if that’s the case, it still doesn’t work for me
In doubt, I did a clean reinstall and reintegrated my devices, everything is well recognized but no data is coming back after the initial addition

4

Indeed, the Z-Wave fixes have passed!

After rereading your other posts, you are using an Everspring key. Maybe it’s coming from the key… I don’t have your hardware, so I don’t know what’s going wrong. I don’t know how to help you more!

5

Hello
my hardware is a raspberry pi3 flashed with the official gladys 4 image (16gb sd card)
I can handle ssh if command returns are needed I can get my hands dirty

the zwave material Zwave - Aide pour la v4 - #49 par Will9

6

Same here, I can only lower my shutter and not raise it (I even deleted and recreated the device)

By the way, I just reconnected to my instance (I wanted to take a screenshot of the Qubino) and no devices at all

No more Hue bulbs, no more Z-Wave, no more cameras, and the OpenWeather key has disappeared Oo (the dashboard was okay but with errors because no devices)

image
image402×383 4.89 KB
image
image398×291 5.46 KB

Edit:

I tried to restore a backup from 4 days ago to avoid having to do everything again.

2020-10-19T18:29:08+0200 <error> index.js:14 (process.<anonymous>) unhandledRejection catched: Promise {
  <rejected> DatabaseError [SequelizeDatabaseError]: SQLITE_CORRUPT: malformed database schema (t_service) - table `t_service` already exists
      at Query.formatError (/src/server/node_modules/sequelize/lib/dialects/sqlite/query.js:423:16)
      at afterExecute (/src/server/node_modules/sequelize/lib/dialects/sqlite/query.js:119:32)
      at replacement (/src/server/node_modules/sqlite3/lib/trace.js:19:31)
      at Statement.errBack (/src/server/node_modules/sqlite3/lib/sqlite3.js:16:21) {
    parent: [Error: SQLITE_CORRUPT: malformed database schema (t_service) - table `t_service` already exists] {
      errno: 11,
      code: 'SQLITE_CORRUPT',
      sql: 'PRAGMA journal_mode=WAL;'
    },
    original: [Error: SQLITE_CORRUPT: malformed database schema (t_service) - table `t_service` already exists] {
      errno: 11,
      code: 'SQLITE_CORRUPT',
      sql: 'PRAGMA journal_mode=WAL;'
    },
    sql: 'PRAGMA journal_mode=WAL;'
  }
}
2020-10-19T18:29:08+0200 <error> index.js:15 (process.<anonymous>) DatabaseError [SequelizeDatabaseError]: SQLITE_CORRUPT: malformed database schema (t_service) - table `t_service` already exists
    at Query.formatError (/src/server/node_modules/sequelize/lib/dialects/sqlite/query.js:423:16)
    at afterExecute (/src/server/node_modules/sequelize/lib/dialects/sqlite/query.js:119:32)
    at replacement (/src/server/node_modules/sqlite3/lib/trace.js:19:31)
    at Statement.errBack (/src/server/node_modules/sqlite3/lib/sqlite3.js:16:21) {
  parent: [Error: SQLITE_CORRUPT: malformed database schema (t_service) - table `t_service` already exists] {
    errno: 11,
    code: 'SQLITE_CORRUPT',
    sql: 'PRAGMA journal_mode=WAL;'
  },
  original: [Error: SQLITE_CORRUPT: malformed database schema (t_service) - table `t_service` already exists] {
    errno: 11,
    code: 'SQLITE_CORRUPT',
    sql: 'PRAGMA journal_mode=WAL;'
  },
  sql: 'PRAGMA journal_mode=WAL;'
}

No feedback in the UI

image
image961×307 6.8 KB

7

Hello! Same as @VonOx, no devices working after the .17 update. No restoration possible from backup. All home automation down.

Given the time it will take to get everything back to normal… for my part, with my little one arriving in a few days, I’m going to take a break for a while.

Good luck to everyone. See you soon.

8

Wasn’t @link39’s PR supposed to fix this issue?

Shit!!

I investigated and found out what caused this. It’s a side effect of the DB migration PR that I merged yesterday from @AlexTrovato.

The PR does:

await queryInterface.removeColumn('t_service', 'enabled');

But in SQLite, it’s not possible to remove a column, it doesn’t exist in SQLite. So Sequelize instead makes a Backup of the table + DROP TABLE + recreates the table without the column + puts the data back.

In the case of Gladys, there is a foreign key between device and service. So you’ll have understood that a DROP TABLE of the service table causes a cascade delete of device, which in turn causes a cascade delete of device_feature, which in turn causes a cascade delete of device_feature_state…

It’s my fault, I should have paid attention to this detail but at the time with the urgency of the bluetooth crash I didn’t see the error.

@VonOx @Terdious, don’t panic, you have backups and your encryption key, so we can recover your backup without any issues, even outside of Gladys.

OK I think I know what’s wrong. Thanks for the detailed feedback with the logs :slight_smile:

Don’t panic @Terdious the backup restoration issue doesn’t seem to be related to the backup itself!

But we need to learn from our mistakes, because this kind of destructive update is just not possible to happen once outside of beta. I’m on it.

9

Ok so I fixed the backup restore issue on master. This is not yet deployed.

After investigating a bit more, the removeColumn command from Sequelize is even more destructive than I imagined, it doesn’t recreate the table correctly, it forgets foreign keys for example…

I think I’m going to add a rule to ban the use of this method in Gladys 4 :warning: :sweat_smile:

Fortunately we saw this now, it would have been hell if it happened in production after the official launch!

10

I made a proposal to remove the ability to use the .removeColumn function of Sequelize.

My PR:

The problem with this is that it is no longer possible to write the « down » migration function of migrations, as we can’t go back.

It’s not crazy, but I don’t think we have a choice, since SQLite doesn’t allow removing a column, we can’t do otherwise (see SQLite Frequently Asked Questions)

@AlexTrovato What do you think?

11

Well, I didn’t quite understand everything, but it’s better now than later.
For my part, I’m still on beta 13. Is that normal?
I also have my OpenWeather box on the dashboard asking me to reconfigure it… yet the key is correct.
I don’t think I’ve seen any Bluetooth or 433 in the betas. Will it be released without them?
I’ll keep my V3 in parallel for now.
Anyway, congratulations and thanks for this huge effort!! :+1:t3:

12

We are currently on beta 17. It’s normal that you are on 13, we changed the Raspbian image since then. The old image is obsolete and will no longer be updated. :warning: After that, don’t change anything for now or you risk encountering the bug above which is not yet fixed! :slight_smile: I advise you to wait until we exit beta :slight_smile: :slight_smile:

Bluetooth has been integrated since this weekend.

For the 433mhz, there is a great PR from @billona on GitHub since June 10th. I requested changes in June, but I think Alexandre must be busy or he didn’t see my message because I haven’t heard back since… :confused: (see https://github.com/GladysAssistant/Gladys/pull/812)

Also, we desperately need testers. No one tests the PRs and it’s complicated to merge something blindly that hasn’t been tested.

It’s a bit of a chicken and egg situation. Currently, no one tests because no one uses Gladys 4 as it’s in beta. But if no one tests, we don’t progress, and we don’t exit the beta… :crazy_face:

Releasing v4 will bring us traffic, new users, more people on the forum, feedback, and with that, we’ll go faster. It’s also more motivating for everyone to know that their work is useful.

I remind you that when I released v3 a few years ago, there were… 0 integrations! :smiley: Integrations came because there was an audience (you) who believed in the potential and helped with development, and this thanks to the launch, the communication around the product.

I think I delayed too much in exiting beta, and we should have exited this beta a long time ago. If we wait for v4 to have all possible imaginable integrations, it will never be released! And that’s a shame because Gladys 4 is a great product :slight_smile:

13

Indeed, Gladys will always evolve, but you are trying to refine your baby… logical :wink:

I will wait then before crushing my current beta.
Anyway, there is almost nothing on it… webcam, weather, and an MQTT test.

Since I am going to run it in parallel with my v3, I would like to test as much as possible according to my abilities (Bluetooth, network detection, 433, …).
However, I don’t know how to retrieve it if it’s not in integration.

14

I’m disappointed to learn that removeColumn doesn’t work but is much worse.
After that, I don’t find it surprising not to go back with the down scripts.
But removeColumn will certainly be indispensable one day or another… we are going to risk ending up with unused columns…

I see in the link that it is therefore recommended to create an alternative table, but with the FK we are going to have fun :slight_smile:

Fortunately, we discover it now indeed.
And sorry for the inconvenience, I stupidly applied my dev habits, I did not expect this behavior.

15

You broke everything @AlexTrovato :stuck_out_tongue:

16

That’s for sure, but as it stands, removeColumn clearly does whatever it wants. I tried disabling the FKs and then putting them back after the migration, and it keeps the data but we still have the problem of FKs/indexes and other things that jump.

After that, it’s not final. Sequelize is preparing a v6 which is currently in beta, maybe they will have improved the behavior on that :slight_smile: :slight_smile:

And anyway, it’s embedded, we don’t change the columns every day, and if there are any unused columns left, well, it’s not « clean » if you’re a purist, but it’s always better than data loss ^^

No problem, you couldn’t have known! I prefer that we saw that now rather than later ^^

By the way, I was thinking it would be cool to put tests on the migrations. Like a generic test in the code that does each migration one by one and checks that after each migration the tables still have the same data and that the FKs haven’t jumped, that will avoid surprises in the future ^^

17

Sorry for the delay on my end, my Mac gave up on me… and so far no sign of life from the little guy, it suddenly turned off and won’t turn on at all :sweat_smile:

18

It’s a sign that you should no longer buy Apple… :sweat_smile::grin:

19

Not bad at all.
It’s clear and not too technical.
In the « From machine learning to the chatbot engine » section, do you specify that it’s for written communication for now, but that voice recognition is planned?
I corrected a few spelling mistakes.
One remark: with double punctuation ( : ; ! ? ), you need 2 spaces, one before and one after. I didn’t change them.
Edit:
Sometimes you write « Gladys », other times « Gladys 4 », or even « Gladys Assistant », but also « Gladys Assistant 4 ». Maybe we should standardize it.

20

I avoid talking about « what is planned », because we are an open-source project that doesn’t have the means to guarantee feature release dates, I prefer not to create false hopes. Let’s talk about what already works :slight_smile:

Thanks!

Well spotted, I’ll go through it again.