Gladys 4 - Arduino service development

[billona]

Hello everyone

In order to make Gladys 4 compatible with my installation (mainly consisting of 433 MHz), I am currently developing the Arduino service.

Integration page:

Arduino service pages:

For now, the interface is simple and not yet finished, but I am working on it. I also need to handle card detection using the existing usb service.

If you want to see the progress of the code, it is available on my repo.

I turn to you to get your opinions on what has already been done, any advice you could give me, or any other discussion on the subject

[peb]

Hello!

This is great news to start the year!
First, I recommend you create a PR on Gladys’s GitHub so everyone knows that Arduino service development is underway!

The question is: what should this Arduino service be capable of? For me, it’s a connector between hardware (Arduinos) and higher-level services (433 MHz…).
This would allow you to say on such an Arduino, such a service is running. (to be seen if it is possible to have n services on an Arduino or only 1).

What do you think?

In any case, you can count me among the testers of Arduino/433MHz services

[billona]

Thank you very much for your reply

@peb

First of all, I recommend you to create a PR on Gladys’s GitHub

The PR is done, we can move forward

@peb

For me, it’s a connector between hardware (Arduinos) and higher-level services (433 MHz…)

I agree with you, my idea is to have 1 device per Arduino, and for each device « sub-devices » representing the services present on the board.

@peb

We need to see if it is possible to have n services on an Arduino or only 1

For me yes, what I was doing with Gladys 3, is that I had different transmitters/receivers connected to my Arduino, and Gladys sent a JSON to the board with the name of the function to call in the board, and other parameters depending on what was requested.

Here is my board with Gladys 3. As you can see I have 2 IR lines (for an LED strip and for a TV), a 433 MHz transmitter and receiver.

So for me it is quite possible to have several services on a single board, we just have to see what is more optimal

What do you think of this solution?

@peb

In any case, you can count me among the testers of the Arduino/433Mhz services

Thank you very much, it’s nice

[Tlse-vins]

For me too,
I use an Arduino with a 433 MHz (433 MHz) transmitter for my TV lift.

[peb]

@billona

For me yes, what I was doing with Gladys 3, was that I had different transmitters/receivers connected to my Arduino, and Gladys sent a JSON to the board with the name of the function to call in the board, and other parameters depending on what was requested.

Good idea the JSON messages! Then we would need to standardize the exchange of messages and define the level of abstraction of the Arduino.

We could have a typical message:

{
  "service": "arduino-433-mhz",
  "message": {
    "value": "xxxx",
    "bbbb": "yyyy"
  }
}

Some random questions I have about creating the service:

For configuration and use, Gladys 4 should be as plug and play as possible. Via the Web IHM, we can identify the connected Arduinos, and then how does it work on the code side?

Does the Arduino already contain the code, and in this case, does Gladys, via a message exchange, identify which services are used for this Arduino?

Do we need to specify the services manually? Should Gladys be able to upload the code to the Arduino? How to code the modules so that they are as easily cumulative in the same Arduino?

Ideally, each sub-module would have a configuration phase, and a standard operating phase.
In the configuration phase, we could specify the necessary pins (the In and the Out for the 433 MHz) so that no modules overlap on the pins.

Here are some random ideas and questions

[pierre-gilles]

Awesome @billona thanks for your message and your development, it will make a lot of people happy

For the serialport/USB connection part, you can base your work on the work that has already been done with the « usb » service of Gladys 4. All connections will go through it, and the routes to list the USB ports already exist.

The goal is indeed to have a service as simple to use as possible. The user plugs in their Arduino, they see it in Gladys, and boom, with one click the user adds it to Gladys and it works.

Indeed, we will need to standardize the message format sent from Gladys to the Arduino and vice versa. For me, this message format could even be the same as that of the MQTT API.

At the end of the day, an Arduino is like a remote service. I was talking about the « Gladys core/Gladys Pod » architecture in the Gladys 4 manifesto, an Arduino can be considered as a Pod that sends information and returns information to Gladys.

See => A privacy-first, open-source home assistant | Gladys Assistant

[Spouet]

Hey Hey!
Great news!
Actually, I’m going to finish my mission this week, and so I thought it might be a good opportunity to tackle this service. But I’m glad you’re on it since I only master C.
On my side, this service is the main part of my Gladys installation. Arduino+433, without it I have no actuator.
And so on my Arduino I have:
Transmitter/receiver 433MHz
A temperature/humidity sensor DHT 22
An air quality sensor
An HC-11 module which I connected the counterpart to my Roomba

I manage all my 433 outlets but also and above all I send « thanks to » JSON all my temperature/humidity information from my various probes, Roomba info etc..
In short, as I say without this service my installation is useless that’s why I haven’t tested Gladys 4 yet.

In short (yes yes again ) if you ever need help I might be able to help.

[JPL]

Excellent: I think this start is promising, I am also waiting for the development of the 433 Mhz. If I understand correctly, the recognition of the Arduino (in my case a nano) will be automated? I should therefore be able to plug in my box with a transmitter receiver and it will be recognized? Then indicate the codes to generate on the devices I will have to create? (living room lighting for example) I may be talking nonsense, being a novice at this level, but it would be nice as a feature
Do you think you can release a test to validate soon (but no, I am not impatient…)
Thanks and follow up then

[Spouet]

Hello,
@billona have you had time to progress on the subject?
Pouet!

[pierre-gilles]

@CamilleB The goal is to develop a unified Arduino API in the same way as MQTT.

Specifications need to be written before starting development.

Let’s see if there are already existing serial communication standards?

It will be a service like any other, to be activated as needed by each user

@CamilleB

I am therefore very motivated to help you if my ideas suit you to create tutorials on Arduinos and test interfaces with Gladys.

Top Don’t hesitate to create tutorials on what you have already integrated into Gladys (MQTT, camera), the Arduino-USB service has not started yet as far as I know…

[camille]

Hello, I am taking back my official account @camille, the account @CamilleB will now be put to sleep

@pierre-gilles We need to write specifications before starting development.

I don’t know what it’s about, but if you have examples and the expected formalism, I can try to move this point forward.

@pierre-gilles Let’s see if there aren’t even existing standards at the serial communication level?

Here’s what I know about serial communications …, if other people want to correct or enrich, I’ll be happy to receive your feedback …

Among the most used « serial » communications by Arduino but also in industry, we can mention the famous RS232 protocol (1960s) which has been and still is in electronics a simple and widespread protocol to use, I²C (1980s) and its little brother 1-wire are also quite used in electronics to address peripherals (memory, A/D, D/A converters, bus expanders …), SPI (1980s) used like I²C in electronics for data exchange with peripherals, LIN and CAN for automotive and industry (not very suitable for our needs in my opinion) and USB (1990s) which needs no introduction and is now a computer standard.

Some examples:

• Temperature/humidity sensors DHT11 and DHT22, 433 Mhz modules use 1-wire
• RFID module RC522, W5100 Ethernet shield and its SD card reader use SPI
• Gyroscope and accelerometer GY-521, real-time clock DS3231, barometer MPL3115A2 use I²C
• Communication/programming with Arduinos use the RS232 link via a USB UART

It is this last link that is used to communicate between Gladys and the Arduino when it is connected via USB.
The other protocols are used by the sensors/peripherals managed by the Arduino.

As a serial standard, I only know the industrial world one, Modbus and its little brothers Profibus, Jbus.
The principle is simple, a master (Gladys) queries or commands slaves (Arduinos).
Messages are composed of the frame for the master: [slave address], [requested function code], [data], [CRC] and for the slave, the response frame if necessary: [slave address], [requested function code], [data], [CRC].

The function code and the CRC comply with a standard.

There are libraries that manage Modbus for Arduinos.

However, we need to program a Modbus master for Gladys, I don’t know if this is easy to do and especially desirable ???

My opinion:

1. This seems like a lot of work for something that might end up being complicated for a lenda user.
   However, I am available to discuss the subject and see if someone else has ideas for existing serial standards to propose or if you think Modbus would be a good idea.

2. I am stuck on the simplicity of the topics used by MQTT. If we reuse the same ideas, it would be enough to broadcast/receive the topics with the data.

On the Gladys side, there is already something and the same on the Arduino side.

The « setup » tab could look like this with auto-detection of the Arduinos you connect

As for the « device » tab, there wouldn’t be much to change (keep the same structure as MQTT)

Same, tell me if I’m delusional or if it’s a good idea.

3. Last point, I developed for one of my old jobs a project in which it was necessary to communicate several slave modules with relatively low data flows. I had used the RS232 support and I had simply used the Modbus philosophy but in a much simpler way. That is, I broadcasted frames with an identifier for each slave, the commands and queries and then each slave answered me.
   There was no CRC because it was not necessary and it was very simple (the proof I could do it )
   So to finish, another solution could be to develop our own standard, what do you think?

@pierre-gilles Top Don’t hesitate to make tutorials on what you have already integrated into Gladys (MQTT, camera)

I look at what already exists and I start …

[pierre-gilles]

@camille Wow, you’ve gone deep into the analysis

When I talked about API specification, the only thing to define is the message format, serial communication is already handled, it’s USB, it works by itself.

Example in Gladys 3: as soon as a new sensor value is received

If you had a temperature sensor with id = 1 (ids were integers in Gladys 3), and the temperature is 20°C, then the Arduino would send the following value via USB:

{
  "devicetype": 1,
  "value": 20
}

So simple

The choice of JSON is debatable. We could adopt another format if there are simpler and lighter standards for an Arduino.

@Terdious you who is an Arduino expert, is the ArduinoJson library heavy to use? Are there alternatives, or standards? (After all, according to the ArduinoJson GitHub, it seems solid and lightweight!)

If we stick with JSON, on Gladys 4, we could imagine a format like:

{
   "message_type": "device.new-state",
   "device_feature": "temperature-salon",
   "value": 20
}

That’s what needs to be defined for each use case, the message format!

[billona]

Hello,

Given the amount of work I’ve had over the past few weeks, I haven’t been able to find the time to focus on developing the service. Now that I’m working remotely, I can finally get back to the code.

Indeed, we discussed with @peb about message formats. We would need to list the different possibilities (433 MHz, IR, screen…). The idea would be to generate the interface based on the user’s hardware.

For example: If the user wants a 433 MHz transmitter, a field appears for them to enter the DATA pin number. Thus, the system would eventually be able to generate an Arduino code adapted to the user and their configuration.

And then, just configure the devices in a classic way. What do you think?

By the way, another feature idea could be to use Arduino with Wi-Fi (ESP8266?) to control hardware remotely.

[camille]

Hello,

Great if you can move the subject forward again, if you need a tester, let me know.

Regarding your example,

@billona For example: If the user wants a 433 MHz transmitter, a field appears for them to enter the DATA pin number. Thus, the system might be able to generate Arduino code tailored to the user and their configuration.

I find it interesting that we can generate generic Arduino code (it goes in the direction that as many people as possible can use GLADYS and its peripherals without having in-depth knowledge). However, where I see a limit or rather a complexity is applying this to a large number of different peripherals and also to the different Arduinos.

But I still find this idea very interesting. If we take the Arduino UNO as an example, in this case, we should try to create a standard mapping with pins that are pre-allocated to functions and the others available as needed. For example: pins D10, D11, D12, and D13 would be reserved solely for SPI, A4 and A5 for I²C, D1 and D2 for USB.

What do you think? Does this reflect your idea?

[VonOx]

@billona do you think managing spaces is interesting with the mqtt service.

@camille I don’t know if you know platformio for vscode. It allows you to write generic code regardless of the type of card. It manages the mapping based on the card you connect.

[camille]

Hello @VonOx, thanks for the info, I didn’t know about Platformio, it looks like a great IDE, debugging was indeed missing in the Arduino IDE, and the help with functions and continuous compilation is appreciated. In short, a real IDE.

Even though the Arduino IDE is developing an Arduino debugger which is currently in the prototype phase…

However, regarding the mapping and generic code, I don’t see where it is in the Platformio IDE (I just installed it and played with it for a few hours last night, but I will continue. If you have any information about it, it would save me time).

But the problem remains, in my opinion, for Gladys, because if we give the user the possibility to choose pins to assign to their peripherals without restricting them, there may be side effects.

What do you think?

[billona]

Hello,

After a restless night of reflection (and coffee ) I was able to make progress on the development. I am therefore writing this small message to describe all the points that came to my mind, I would like to have your opinion on the progress of this service.

1. How does the front-end look?

I imagined 2 pages:

• Setup

On this page, we will have the listing of USB ports, and we will be able to indicate on which port a card is connected to be able to register it, and make it pass to the devices page.

• Device

Each device will correspond to a card. Each card will be equipped with « sub-devices », which will represent the equipment connected to the card (433MHz, IR, …)

2. The Arduino code

I thought about this for a long time because it’s a point not to be neglected. It will be necessary that each user has THE correct Arduino code. That’s why I thought of a « universal » code specific to the service. Thus, the more equipment we consider, the more important the code will be. The specificity is that each equipment will have its « sub-service », and that this sub-service will call a single piece of code, which lightens the load of the card.

3. The standardization of JSON to send to the card

To communicate, we absolutely need a standard. It will therefore be as follows:

{
    "service": "the service of the sub-device",
    "message": {
        "DATA_PIN": "the data pin",
        "code": "The code to send"
    }
}

For the moment, I will focus mainly on the « transmission » modules, as they seem to me the simplest to start with for a first result.

4. The current state of development

I am of course still developing, but my code is always available here, if someone wants to give me their criticism

@camille

For example: pins D10, D11, D12, and D13 would be reserved exclusively for SPI, A4 and A5 for I²C, D1 and D2 for USB.

What do you think? Does this reflect your idea?

Actually, I had more in mind to directly ask the user the type of their card, and the pins in question, to generate the code based on that. But I think the question no longer arises if everything is okay to start with a universal code.

Thanks in advance for your feedback

[camille]

@billona

I’ll look at the code and get back to you.

I’m 100% in agreement with the idea of a universal code that allows managing the interface with Gladys, in my opinion, it’s the best solution.

For your idea of asking for the type of Arduino card, it’s a good idea, I think it would be in the « device » window?
If I understand correctly, in this tab, we would choose the type of connected equipment (433 MHz, …) and the type of Arduino?

In this case, the pins related to the com would be automatically reserved by the system

For the Arduino Settings view, I also think it’s the right solution to be able to choose via the USB port, but when you connect an Arduino to the Raspberry Pi, is there a particular indication that differentiates the USB port from the others? Or in other words, if I connect an Arduino and another device, is there something that allows you not to make a mistake and to correctly choose the Arduino’s USB?

[billona]

@camille

For your idea of asking for the type of Arduino card, it’s a good idea, I think it would be in the « device » window?
If I understand correctly, in this tab, we would choose the type of connected equipment (433 MHz, …) and the type of Arduino?

Yes, I had something like that in mind, we’ll see how to organize it.

@camille

In this case, the pins related to com will be automatically reserved by the system

There I would say to be careful with the term « automatically ». My initial idea is to leave the choice to the user. He cables his equipment as he wants. For example, he can choose to put the 433 data pin on pin 12… Or 13, it’s up to him to decide.

In any case, this way, after he has made his choice and connected his Arduino, every time he adds equipment, he is asked to enter the pin he has chosen. Thus, when the Arduino code function is called, the signal to be sent will be given to him as a parameter, but also the pin on which to send it. This makes the code a bit more global

@camille

For the Arduino Settings view, I also think it’s the right solution to be able to choose via the USB port, but when we connect an Arduino to the Raspberry Pi, is there any particular indication that differentiates the USB port from the others? Or to put it another way, if I connect an Arduino and another peripheral, is there something that allows you not to make a mistake and to choose the Arduino’s USB?

Currently no. To list the USB ports, I go directly through the USB service, and this one for the moment only provides one piece of information: comPath. For the moment, I do it manually: I refresh the list and I look at the port that is added when I connect my Arduino.

After that, I made the choice to say that there will be only one Arduino connected to the Raspberry Pi. This way, I can store the USB path of the card in the t_variables table of the DB without ambiguity. After that, we can always think about adding more via Wi-Fi as explained previously, but for a starting point I found it simpler to do it that way. What do you think?

In any case, thank you for your support! I’m really looking forward to having a first functional version

[Reno]

Hello,

I saw your various exchanges on this thread and it made me wonder about the need to use an Arduino when the Raspberry Pi is capable of managing all these peripherals on its own.

So, I think I must have missed something. Could you tell me more?

One of my ideas (for later, when I have the time) would be to create a generic Python service that would allow addressing the Raspberry Pi’s inputs and outputs and communicating with Gladys via MQTT.
I get the impression that this is somewhat similar to what you are already doing, right?