I recently added the ability to select an AI model in Gladys, and I’d like to get your feedback on the topic: which model do you find the most intelligent for home automation use (understanding requests, relevance of responses, etc.)?
Feel free to test multiple models and vote for your favorites (max 3 choices). Your feedback will help me decide which ones to highlight by default in Gladys!
I tried in Auto but we don’t know which one is taken: the cheapest? the fastest?
The results weren’t great for the same question asked, even when changing with the top three:
But precisely, I have feedback that for some people it isn’t so intelligent, surely it gets a bit stupid as the context increases. I’m looking for a new default model
From my side, I’ve conducted several real-world tests on my current setup, which includes lights, shutters, outlets, switches, and cameras. The one that performs the best is Qwen 3.6 35B (60% success rate compared to 10% for Mistral), but it’s far from perfect compared to Alexa+ or Google, which understand over 95%! But this is just my opinion and doesn’t commit anyone else!
Not at the moment, but if I had to choose a model, I might as well choose a French one.
I use Mistral on a daily basis and I’m satisfied with the results.
As for the others, apart from Llama and Gemma, I don’t know them.
Ok, because for this Mistral 3.2 Small is quite limited and @Chris75, who does a lot of testing and uses AI a lot in Gladys, complains about it.
For the French side, in any case these models are Open-Weight, and hosted in France at Scaleway, so it doesn’t matter whether the model is French or not, these are « free » models, I only pay for the GPUs. The important thing is really the quality of the result
I’m interested in your debug file to analyze the different tests you’ve performed. You can download it in one click from the Artificial Intelligence integration:
By the way, I just modified the Auto mode to now use Qwen 3.6 by default. I find it significantly more performant than Mistral Small 3.2, which was starting to show its limits on certain requests.
On my end, I also find that qwen3.5-397b-a17b is excellent. It’s one of the most expensive models on the list, but the quality of the responses is really up to par. We’ll have to see, with use, how it compares to Qwen 3.6.
A little test, worth what it’s worth (just one case, not a scientific study ), but it really helped me understand the differences between models.
Why test a scene specifically? Because that’s clearly what holds me back the most in daily use. A simple command (« turn on the kitchen »), any model can handle it. But a scene is often more complex and longer to describe, so it’s more revealing of whether the model actually reasons or just bluffs.
I asked: « create a scene that turns off the kitchen ceiling light automatically 30 seconds after it has been turned on », without giving the exact device name in the prompt (the real name is Ceiling Light 3 Bulbs, Kitchen room). And that was intentional, because in real life, my partner, for example, doesn’t know the technical names of the devices, she just describes what they do or where they are (« the kitchen ceiling light », « the lamp next to the sofa »…). An assistant that needs the exact name to work isn’t usable for her.
The Crash Test by Model:
Mistral Small and Holo2-30B: died in combat, the API crashed every time. Nothing to get from them.
Gemma 4 and Gemma 3-27B: they tried to call scene_create, but they made things up (fake device_feature, non-existent icon) → boom, error 422. Gemma 3-27B at least had the good reflex to check the state first (device_get_state), but failed on the naming.
Qwen3.6: error on creation, then the API crashed too.
Pixtral 12B: the worst in « I’m just pretending » mode. It keeps looping the JSON saying « I’m going to execute it now » but never actually calls the tool, even when you tell it « go ahead », « call the function » three times in a row.
Llama 3.3-70B: good reflexes on the logic (delay + turn-off), but invalid device_feature and icon → 422 direct.
Qwen3-235B: the sneakiest of the bunch. It tells you « the scene has been created successfully » when it called no tool. Zero tool call, pure success hallucination. The worst case for me, the silent false positive that makes you think you have a car when you have nothing.
Qwen 3.5 (379B): the only one of the nine that managed to find the right device on its own. It first checks the state of the devices in the Kitchen room (device_get_state), identifies the right ceiling light from its natural language description, then sends a scene_create that goes through directly. The scene actually exists behind it, tested and verified.
What convinced me: it’s not just about outputting JSON that looks good, it’s the ability to find the right device from a natural description rather than demanding the exact name or improvising a random ID. That’s exactly the kind of friction that makes me always put off creating my scenes — if I have to know the technical name of every device by heart for it to work, I might as well do it manually in the interface. An assistant that understands « the kitchen ceiling light » without any further detail really changes the game for adoption by the whole family, not just me.
Thanks for this very comprehensive analysis, it’s a huge help!
In the end, your feedback is quite consistent with the technical capabilities of each model. Qwen 3.5 (379B) is the largest model in the selection, and in practice, it’s also the most performant.
@Chris75, I’d be curious to hear your thoughts on Qwen 3.5 (379B)!
I’m also considering adding other models, this time hosted on DigitalOcean in Germany, as Scaleway’s offering is still quite limited on the most powerful models. I’m particularly thinking of DeepSeek V4 Pro, which offers an excellent quality/price ratio and is available on DigitalOcean.
We’d still be on a European hosting, on a private cloud infrastructure.
Would this compromise seem acceptable to you?
I would say yes for my part: Europe, GDPR (normally followed by Europeans), to be seen on the terms and conditions and the hosting of their servers.
Have you checked with our Swiss colleagues at Proton and Infomaniak?
On my side, I haven’t tested everything yet, I’ve mostly had a lot of AI crashes, I’ll share my logs with you in PM. In any case, the AIs struggle with my pool
Otherwise, which AI do you use for the weekly summary? Because this one works very well (always in relation to my pool consumption and cost).
Well, I’ve done a lot of testing with Qwen 3.5 397B a17b for everyday uses (lighting, roller shutters, switches, sensor status/history, power outlet consumption, EDF consumption, cameras) and the results are indeed better than with Qwen 3.6 35B.
I haven’t tested scene creation.
Main issues identified: the AI can’t adjust the brightness/color of lights (that’s a major shortcoming), sometimes can’t decipher histories, can’t find weather data, can’t find battery status, etc.
Thanks for your tests @Chris75, it’s super useful!
I just switched the Auto mode to Qwen 3.5 397B A17B. That way, whether in chat or voice, the responses should be more relevant. I’ll monitor the costs to see if it’s viable to use it as the default model.
However, for the weekly digest, I’m sticking with Mistral Small 3.2. With Qwen 3.5, the prompt becomes so large that the model takes more than 2 minutes to respond, and in the end, I don’t get better results than Mistral according to my tests.
QUESTION: « What was the power consumption in kWh of the inverter outlet for the day of July 12, 2026? »
RESPONSE: According to the Inverter Outlet data, I see the cumulative consumption history in kWh. However, the returned data is truncated, and I cannot see the exact values at the boundaries of July 12, 2026 (midnight on the 12th and midnight on the 13th).
What I can observe:
July 10 around 12:30 p.m.: 48.99 kWh (cumulative)
July 11 around 11:45 p.m.: approximately 50.08 kWh (cumulative)
To obtain the exact consumption for July 12, you would need to subtract the cumulative value at the end of the day on July 11 from that at the end of the day on July 12. Unfortunately, the precise data for this period is not fully visible in the result.
Would you like me to check the data in another way or for a different period?
Good news: Gladys’ AI is going to become much faster, and at the same time more relevant.
The problem
With each new feature, we add tools that the AI can use (turn on a light, read a temperature, create a scene…). Except that all these tools are sent to the model with every message. Result: the prompt becomes heavier, and even a simple “turn on the living room light” becomes a complex request to analyze. This is a known phenomenon: the more tools available, the more the precision of selection degrades, regardless of the model used.
The solution: a two-step routing
Instead of a single big call, we make two:
Classification: a light and fast model receives only the list of tools and determines the type of request.
Execution: the real query goes with only the useful tools. The heaviest tool, scene.create, is only loaded if the request has been classified as a scene creation.
This classification step brings a second benefit: it allows disabling the model’s “thinking” mode on simple requests, where it often added several seconds for nothing. Scene creation, on the other hand, keeps this reasoning mode because it really needs it to remain reliable.
Concretely for you
On a common command (light, temperature, shutter…), we go from 10 to 15 seconds in the worst cases to just a few seconds. The AI part of the chain even goes below a second.
This will be available in the next version of Gladys!
)
), but it really helped me understand the differences between models.
