Let's talk about Gladys V4

[pierre-gilles]

Once again I am very surprised by the power of Sequelize!

Let’s say I want to retrieve all the « locations » with their associated user.

I do:

I get:

But what’s interesting is not that! It’s how Sequelize does this under the hood.

Some ORMs in this kind of situation make X requests:

• A SELECT * to fetch all the rows
• Then a SELECT per foreign key to fetch the nested attribute

Here, Sequelize makes a single request

SELECT `t_location`.`id`, `t_location`.`user_id`, `t_location`.`latitude`, `t_location`.`longitude`, `t_location`.`altitude`, `t_location`.`accuracy`, `t_location`.`created_at`, `t_location`.`updated_at`,
`user`.`id` AS `user.id`, `user`.`firstname` AS `user.firstname`, `user`.`lastname` AS `user.lastname`, `user`.`email` AS `user.email`, `user`.`birthdate` AS `user.birthdate`, `user`.`language` AS `user.language`, `user`.`password_hash` AS `user.password_hash`, `user`.`role` AS `user.role`, `user`.`created_at` AS `user.created_at`, `user`.`updated_at` AS `user.updated_at`
FROM `t_location` AS `t_location`
LEFT OUTER JOIN `t_user` AS `user` ON `t_location`.`user_id` = `user`.`id`;

Then it reassembles the nested object using its « table.attribute » format in the « AS »

It’s super performant!

[pierre-gilles]

@piznel

I didn’t see them in the model. Will they be defined later? Or is the analysis outdated?

As I mentioned earlier, it was work in progress, I just pushed the indexes

@piznel

For my part, with a « non-dev » reasoning, a feature is defined with a category (like « light »), then a sub-category (like « dimmable »).
In this case, we would have a pair « light/dimmable », which allows defining the expected display and behavior, and the « type » would disappear in this table.

Ok I understand the idea!

It’s more a question of naming in fact + a finesse in the granularity of the types in fact.

I like the idea of quite fine types.

What I don’t understand about you, take your example for example:

category: "light"
sub-category: "dimmable"

What does that mean to you? In the UI what do we display?

After that, I like the idea of types more fine than just « binary », « multilevel » and « color ».

If in the idea we take a quite general category + a sub-category (to be seen for the naming, I find that naming it « type » is more coherent), what do you think of this schema? (I just drew it)

(sorry if it’s hard to read, from left to right « light », « sensor », « switch », « fan », « door lock »)

What we just need to do after that, is that for each pair (category/type) we have a function that returns the kind of button to display, but it’s not very complicated.

[pierre-gilles]

You didn’t say what you thought about my schema!

I admit I don’t fully understand everything here ^^

Isn’t « Parameters » redundant with « subcategory »?

To go back to our modeling to get off on the right foot.

Are we in agreement that:

• A Device is a physical device, an object.
• A Feature (which we call « DeviceFeature » to make it more easily understandable in the DB, but in short, it’s a feature), is a feature of a device.

?

Example:

My « Philips Hue Bulb 1 » device has 3 features:

• On/Off
• Adjust brightness
• Adjust color

So we could represent this device:

{
  "name": "Bulb 1",
  "service_id": "1ec7966f-e3ef-47d7-9aef-083a59956a68",
  "features": [{
      "category": "light",
      "type/sub_category": "binary"
    },
    {
      "category": "light",
      "type/sub_category": "brightness"
    },
    {
      "category": "light",
      "type/sub_category": "color"
    }
  ]
}

There’s an idea in your Excel, but could you give me some examples of what you see? (in the way I did here).

I don’t understand the hierarchy and the logic behind each entity/attribute you present to me.

I like the notion of parameters which is very useful for Z-wave, but how do we store that?

I remember we already talked about this with @MathieuA, it could be a « parameters » boolean attribute in a DeviceFeature which would allow to say if a DeviceFeature is a parameter or a real feature.

[pierre-gilles]

Hello everyone!

I continue my reflections, this time on the code style that will have the script/see the core and the modules part.

1/ First approach, very object-oriented:

Controlling a device:

const kitchenLight = await gladys.device.findOneBySelector('kitchen-main-light');
await kitchenLight.turnOn();
await kitchenLight.setBrightness(10);
await gladys.helpers.wait(100);
await kitchenLight.setBrightness(20);

Controlling a set of devices:

const kitchenLightGroup = await gladys.device.find('kitchen.light'); // ROOM_SELECTOR.CATEGORY
await kitchenLightGroup.turnOn();
await kitchenLightGroup.addBrightness(10);

2/ Second approach, more functional like we have currently:

await gladys.device.turnOn('kitchen-main-light');
await gladys.device.setBrightness('kitchen-main-light', 10);
await gladys.helpers.wait(100);
await gladys.device.setBrightness('kitchen-main-light', 20);

await gladys.device.turnOn('kitchen.light');
await gladys.device.addBrightness('kitchen.light', 10);

Small analysis

The two approaches are very different and have very different implications.

1/ In the first case, after receiving the data from the database, we will have to build these objects according to each category of device, and add this entire object layer. It’s quite close to the approach you had put on this channel @piznel with the abstract-things repo.

I like visually what it gives, it looks clean and it’s quite sexy to attract devs. So on the syntax and communication side it’s a plus for me.

After my concern is that we want to extend this logic to many places, and it can be a burden where we would like a soft rather light and easy to use. I’m not convinced that an object approach makes sense in a backend. A backend for me is above all just a bridge that does CRUD between the DB and the frontend, not necessarily need to create objects that can be heavy in memory and bring development complexity.

2/ I like less visually what it gives, it’s less sexy that’s for sure.

After we clearly know what’s happening, and in terms of memory management the job is given to the core: we call the function, the function does its job, and that’s it.

Conclusion

I don’t know if it’s a choice between one or the other, we can have a hybrid approach.

I would like to have your opinion, it’s clearly a nice little debate

[pierre-gilles]

Well I disagree on that point, we would be storing too specialized and duplicated information.

1/

light_dimmable: I don’t understand the point of having this. You store 2 pieces of information in one field: that it’s a device that manages light + that this device is dimmable. On that point, I refer you to functional dependencies in relational databases

If I ask Gladys « Turn on the living room light », Gladys needs a way to identify all devices that manage light (whether it’s dimmable or not doesn’t matter in this question, it’s secondary information).

2/ On the other hand, the question of bringing the category up to the device is a real question. For now it’s in DeviceFeature because some devices (I’m thinking of Z-Wave in particular) do a bit of everything, and so we put the category in the deviceType to manage these « hybrid » devices.

Then the question arises. If you have a surveillance camera that also acts as a temperature and motion sensor? If you have a door that also reports temperature and humidity data? How do we classify them?

If you have examples of strange devices, I’m interested!

Personally, I’m in favor of bringing the category up to the device, it’s clearer.

[pierre-gilles]

I think the best way to define all this is to look at real cases!

@piznel

The Xiaomi gateway, which also acts as a light sensor, motion detector, HP, and light!

Excellent case!

So, what category would this device fall under if we consider that the category is included in the device?

1/ I’m not a fan of splitting a device into 3 devices, it’s not logical according to our definition « a device is a physical peripheral ».

2/ If we give it a category like « bridge » for example.

A « binary » type deviceFeature no longer makes sense: is it to turn the gateway on/off? Or to turn on the gateway’s light?

So, we come back to needing to add an additional category to deviceFeature to indicate « this feature is for light management ».

PS: When I told you that the modeling part is 80% of the work, I wasn’t joking The code is nothing!

[Albenss]

In the « weird » category, there’s also the Fibaro FGMS-001 (in Z-Wave) which acts as a motion detector, temperature sensor, light sensor, and vibration sensor, I think

[pierre-gilles]

Mmm ok

Too bad, I liked the idea of putting the category on the device, but it restricts us too much…

So we agree, we keep the category + the type on the DeviceFeature?

I admit that in terms of naming, I prefer « type » to « sub-category ».

I agree that it’s cool to have more detailed types though!

So instead of having:

• binary
• multilevel
• color

We would have richer types:

• binary
• brightness
• temperature
• color

Like my schema above! Quick reminder:

[pierre-gilles]

Another point we haven’t discussed is the famous device/deviceFeature parameters!

Z-wave has device-specific parameters, and currently, these are mixed with deviceFeature (not great).

We could add an attribute in deviceFeature to indicate that the feature is a parameter.

Problem: if the parameter is a string, how do we handle it?

Example: if we want to store a camera’s URL in Device/DeviceFeature, for example.

This could be another table…

Another unrelated idea: Add a boolean in DeviceFeature to enable or disable the historical tracking of values (very practical for certain detectors that send values and for which we don’t necessarily want to track the values historically).

(I’m dumping my ideas like this)

[pierre-gilles]

Small model of what this could look like:

[pierre-gilles]

@piznel

a JSON in a param field? That could also be useful for others.

I thought about that but it doesn’t work in the case of z-wave! It sends the parameters one by one as if they were features, and so you would need to perform a heavy operation of:

• SELECT Device
• Add the value in the JSON
• UPDATE device

Whereas here it’s in one operation!

And above all, I prefer the relational aspect for everything that is exposed to developers, it must be strict in the DB otherwise it’s a mess in 30 seconds ^^

The rare places where we will do JSON in the gladys DB is for very specific reasons!

[VonOx]

I don’t have much of an opinion because I don’t have much experience in all of this, but your reflection is very interesting, guys.

[pierre-gilles]

This morning I worked quite a bit on the topic of « modules » which become « services » in Gladys 4.

I wrote documentation and coded an example service from start to finish (including tests), to show what a service looks like in Gladys 4.

This is just a first draft, I will surely make revisions based on your feedback

The README and the code are available here →

https://github.com/GladysAssistant/gladys-4-playground/tree/master/server/services/example

@VonOx

I don’t have much of an opinion because I don’t have much experience in all of this, but your reflection is very interesting, guys.

Thanks @VonOx!

[pierre-gilles]

Well, I’ve switched to wireframe mode and worked on the onboarding process.

So far, I’ve identified 6 steps:

1/ Welcome: Gladys welcomes the user. A quick reminder of the project’s philosophy, values, and history.

2/ Local account creation: name, email, password & password repetition

3/ Home configuration: Recording the name of the house and adding rooms

4/ Adding devices: The user adds their devices. They have a search box and can select the integrations they want to configure. (Optional)

5/ Notification configuration: To stay in touch with Gladys, Gladys offers to configure communication services: Telegram, SMS free, etc.. (Optional)

6/ Connection to the Gladys Gateway account: Presentation of the Gateway, and connection. (Optional)

If you think of other steps, don’t hesitate!

[pierre-gilles]

Hello everyone!

I hope you had a good weekend.

Yesterday and this morning I worked on defining the states / state transitions. This may seem like a trivial subject, but it’s a real reflection.

Each entity in Gladys is a finite state machine with several states and transitions between each state. These states and transitions need to be defined and named.

User sleep

4 states:

• Awake
• Asleep
• Should wake up
• Should go to bed

Here is the transition diagram. Each transition corresponds to an event in Gladys, a « trigger » on which any scene can be connected.

Home alarm

In Gladys 4, I am introducing the concept of a home alarm so that Gladys can finally be a clear and proper alarm center.

Like sleep, 4 states:

• Armed
• Disarmed
• In the process of arming
• In the process of disarming

User presence at home

Scene execution

Sun state

Work presence

Finally, an example of a device in the « light » category

If these examples speak to you / make you react, don’t hesitate, nothing is set in stone

[VonOx]

Yes you’re talking about scenario, the graphs are the Flow between the states.

And there is indeed the armed state for the alarm.

[pierre-gilles]

@Jap93

For the home alarm part, how do you handle the « alarm activated » state after a detection? Shouldn’t there also be an « alarm activated » state?
For the home entry/exit state, will this automatically activate/deactivate the alarm or do you need to go through a scenario?
For now, I only see that because you’ve thought about most cases

Actually, in this diagram, I had assumed that the « activation » part of the alarm was more on the scenario side. However, I think you’re right, it would be worth putting it there, at least to define the names of the different transitions

Thanks for the feedback!

[pierre-gilles]

Hello everyone!

Today, a lot of work on the frontend, I’m in the thick of it Here’s a little preview, any feedback/ideas for the navigation bar organization I’m all for it!

For the login I’m hesitating between two versions…

Let me know what you think!

[Hamtaro]

With logo for login for me.
By the way, you don’t rename Gladys Assistant in the dashboard either?

[pierre-gilles]

@Hamtaro

With a logo for the login for me.

I think I’ll go with that, everyone on Twitter is unanimous

@Hamtaro

By the way, are you not renaming Gladys Assistant in the dashboard as well?

Well spotted!

New page, integrations:

@piznel

I’m working on a Homekit module at the moment.

Hello @piznel! Great stuff!

@piznel

I don’t see what the feature-specific parameters for Z-Wave are precisely. Do you have an example?

Each Z-Wave device has parameters (e.g., refresh rate, light sensor sensitivity, intruder detection threshold, etc.) that you can adjust. I proposed a DeviceParam to store these parameters, which, for me, are not really features, they are parameters.

This DeviceParam could also be used to store other things, for example: the credentials of an IP camera, or something else.

Do you see the idea?

@piznel

I made an excel listing all the services and characteristics of Homekit.

There you go Philippe, the machine is back

@piznel

It seems like a good starting point. What do you think?

It’s close to what we discussed, right? A category + a type?

With a few differences in naming, but the idea is the same!

@piznel

I think it would be good to define each property of the categories; this would also standardize the display; in the example above, the color picker cannot work with the Milight module.

I’m 100% agree to harmonize!

We need to make a choice that covers a wide range to not lose information on devices that allow fine values, and we will need to convert the color before saving it in Gladys.

@piznel

0 to 65535 for the Hue module,

The Hue module is complicated, you have 3 deviceType: hue and saturation and brightness, it’s HSL if I’m not mistaken.

Milight I have the impression they do RGV but there are HSL conversion functions in this module for example => https://www.npmjs.com/package/node-milight-promise (quick search)

It’s up to us to decide which format we prefer in Gladys, and to code the different getters/setters in Gladys 4 so that the module can just send its data