Instance Gladys KO

qleg · April 9, 2024, 8:32pm

Hello,

My Gladys instance was down this morning.
The Gladys container is running but does not respond on port 80.
All my other containers on my mini-PC are working (node-red, z2m…)
I have some errors in the gladys logs that appear quite frequently: sudo docker logs Gladys

2024-04-09T04:50:48+0200 index.js:16 (process.) TypeError: Invalid value « undefined » for header « authorization »
at ClientRequest.setHeader (node:_http_outgoing:651:3
…
code: ‹ ERR_HTTP_INVALID_HEADER_VALUE ›

I have the impression this is since an update triggered by watchtower
sudo docker ps :

« 23 hours ago » indeed corresponds to the moment when I stopped receiving Gladys Telegram notifications.

What have I done recently on Gladys?
→ I configured a scene with a ‹ Wait › action of 24h (I doubt it comes from that)

Of course I tried restarting the gladys container as well as rebooting the entire machine but the problem persists. Gladys is not accessible from other devices.
I can share more logs if needed.

Do you have any other debugging leads? Am I the only one experiencing this behavior since an update?

Thanks

pierre-gilles · April 10, 2024, 7:27am

Hi @qleg!

Do you have the full log?

Monday’s upgrade was really minor, on the backend almost nothing changed… So I’d be surprised if it was the upgrade itself.

A few routine checks:

Is there enough disk space left on your mini-PC? df -h to check in the CLI
Is the mini-PC not under heavy load in some way? htop is handy to check that

qleg · April 10, 2024, 8:16am

Hello,

Yes, I checked the disk space: no problem, there’s still plenty of room.
And htop is fine too; my machine isn’t maxing out its resources.

I’ll send the full log as soon as possible.

Thanks

qleg · April 10, 2024, 5:44pm

Here is the full log :

2024-04-09T04:50:48+0200 index.js:15 (process.) unhandledRejection catched: Promise {
TypeError: Invalid value « undefined » for header « authorization »
at ClientRequest.setHeader (node:_http_outgoing:651:3)
at new ClientRequest (node:_http_client:291:14)
at Object.request (node:https:366:10)
at RedirectableRequest._performRequest (/src/server/node_modules/follow-redirects/index.js:284:24)
at new RedirectableRequest (/src/server/node_modules/follow-redirects/index.js:66:8)
at Object.request (/src/server/node_modules/follow-redirects/index.js:523:14)
at dispatchHttpRequest (/src/server/node_modules/@gladysassistant/gladys-gateway-js/node_modules/axios/lib/adapters/http.js:202:25)
at new Promise ()
at httpAdapter (/src/server/node_modules/@gladysassistant/gladys-gateway-js/node_modules/axios/lib/adapters/http.js:46:10)
at dispatchRequest (/src/server/node_modules/@gladysassistant/gladys-gateway-js/node_modules/axios/lib/core/dispatchRequest.js:53:10)
at Axios.request (/src/server/node_modules/@gladysassistant/gladys-gateway-js/node_modules/axios/lib/core/Axios.js:108:15)
at Axios. [as get] (/src/server/node_modules/@gladysassistant/gladys-gateway-js/node_modules/axios/lib/core/Axios.js:129:17)
at Function.wrap [as get] (/src/server/node_modules/@gladysassistant/gladys-gateway-js/node_modules/axios/lib/helpers/bind.js:9:15)
at Object.get (/src/server/node_modules/@gladysassistant/gladys-gateway-js/lib/request.js:86:6)
at GladysGatewayJs.enedisGetDailyConsumption (/src/server/node_modules/@gladysassistant/gladys-gateway-js/index.js:1177:23)
at Gateway.enedisGetDailyConsumption (/src/server/lib/gateway/enedis/gateway.enedisGetDailyConsumption.js:17:56)
at recursiveBatchCall (/src/server/services/enedis/lib/enedis.sync.js:24:37)
at /src/server/services/enedis/lib/enedis.sync.js:92:32
at tryCatcher (/src/server/services/enedis/node_modules/bluebird/js/release/util.js:16:23)
at Object.gotValue (/src/server/services/enedis/node_modules/bluebird/js/release/reduce.js:166:18)
at Object.gotAccum (/src/server/services/enedis/node_modules/bluebird/js/release/reduce.js:155:25)
at Object.tryCatcher (/src/server/services/enedis/node_modules/bluebird/js/release/util.js:16:23)
at Promise._settlePromiseFromHandler (/src/server/services/enedis/node_modules/bluebird/js/release/promise.js:547:31)
at Promise._settlePromise (/src/server/services/enedis/node_modules/bluebird/js/release/promise.js:604:18)
at Promise._settlePromiseCtx (/src/server/services/enedis/node_modules/bluebird/js/release/promise.js:641:10)
at _drainQueueStep (/src/server/services/enedis/node_modules/bluebird/js/release/async.js:97:12)
at _drainQueue (/src/server/services/enedis/node_modules/bluebird/js/release/async.js:86:9)
at Async._drainQueues (/src/server/services/enedis/node_modules/bluebird/js/release/async.js:102:5)
at Immediate.Async.drainQueues (/src/server/services/enedis/node_modules/bluebird/js/release/async.js:15:14)
at processImmediate (node:internal/timers:476:21) {
code: ‹ ERR_HTTP_INVALID_HEADER_VALUE ›
}

I don’t know if it’s related to my crash but I’m trying to find some information

qleg · April 10, 2024, 6:44pm

On restarting the Gladys container, these logs catch my attention:

2024-04-10T20:40:42+0200 index.js:15 (process.) unhandledRejection catched: Promise {
TypeError: Cannot read properties of undefined (reading ‹ substr ›)
at /src/server/lib/scene/scene.addScene.js:45:47
at Array.forEach ()
at SceneManager.addScene (/src/server/lib/scene/scene.addScene.js:36:20)
at /src/server/lib/scene/scene.init.js:22:10
at Array.map ()
at SceneManager.init (/src/server/lib/scene/scene.init.js:20:30)
at Object.start (/src/server/lib/index.js:150:9)
at /src/server/index.js:51:3
}
2024-04-10T20:40:42+0200 index.js:16 (process.) TypeError: Cannot read properties of undefined (reading ‹ substr ›)
at /src/server/lib/scene/scene.addScene.js:45:47
at Array.forEach ()
at SceneManager.addScene (/src/server/lib/scene/scene.addScene.js:36:20)
at /src/server/lib/scene/scene.init.js:22:10
at Array.map ()
at SceneManager.init (/src/server/lib/scene/scene.init.js:20:30)
at Object.start (/src/server/lib/index.js:150:9)
at /src/server/index.js:51:3

I doubt this is related to a scene I recently created with a 24h « Wait » action; we’d have more logs than this, wouldn’t we?

pierre-gilles · April 11, 2024, 5:20am

Ah indeed the second log is much more interesting! That’s what’s causing the problem!

Here’s the line that’s crashing:

github.com/GladysAssistant/Gladys

server/lib/scene/scene.addScene.js

bcdc5eefb


      
          if (scene.triggers && scene.active) {
            scene.triggers.forEach((trigger) => {
              // First, we had a trigger key, import to uniquely identify this trigger
              trigger.key = uuid.v4();
              if (trigger.type === EVENTS.TIME.CHANGED && trigger.scheduler_type !== 'interval') {
                const rule = {};
                rule.tz = this.timezone;
                switch (trigger.scheduler_type) {
                  case 'every-month':
                    rule.date = trigger.day_of_the_month;
                    rule.hour = parseInt(trigger.time.substr(0, 2), 10);
                    rule.minute = parseInt(trigger.time.substr(3, 2), 10);
                    rule.second = 0;
                    break;
                  case 'every-week':
                    rule.dayOfWeek = trigger.days_of_the_week.map((day) => nodeScheduleDaysOfWeek[day]);
                    rule.hour = parseInt(trigger.time.substr(0, 2), 10);
                    rule.minute = parseInt(trigger.time.substr(3, 2), 10);
                    rule.second = 0;
                    break;
                  case 'every-day':

You must have created a scene that triggers every month, but you didn’t set the hour and it’s causing Gladys to crash.

While the bug is fixed and pushed to Gladys, can you connect to the DB, find the scene in question, and delete it?

qleg · April 11, 2024, 7:02am

ok thanks for the info, I’ll see what I can do

qleg · April 11, 2024, 4:23pm

Indeed, it was caused by a scene scheduled monthly without a time (my oversight).
I deleted the scene from the database and my instance is working correctly, thanks

Some info on the bug for a fix:

We do get an error message when saving with a monthly trigger without the time: « An error occurred while saving your scene. Please check that all actions / triggers are filled in and correct. » but it still saves the scene and you can leave the scene edit page without any problem.
The instance crash occurs when the scene runs in scheduled mode
We could set a default time if not provided (for example 00:00)
Or block exiting the scene edit if there’s an error, but that seems a bit more complex to implement

Terdious · April 15, 2024, 10:29am

Possible solution : be able to save scenes that are in error, but prevent them from being activated.
=> Error message when saving and on each

pierre-gilles · April 15, 2024, 3:27pm

Thanks for the investigation, can you create a GitHub issue to keep track of the bug?

qleg · April 15, 2024, 6:58pm

Yes, I’ll create a GitHub issue: Crash instance at reboot when a scene is configured every-month without hours · Issue #2052 · GladysAssistant/Gladys (github.com)

Fix:

pierre-gilles · April 19, 2024, 2:46pm

Hi @qleg, I worked on this since it’s really ultra-critical!

I opened a PR:

github.com/GladysAssistant/Gladys

Fix #2052: Prevent crash with scheduled trigger

master ← fix-crash-scheduled-trigger

ouvert 02:22PM - 19 Apr 24 UTC

Pierre-Gilles

+73 -3

### Pull Request check-list To ensure your Pull Request can be accepted as fa…st as possible, make sure to review and check all of these items: - [x] If your changes affects code, did your write the tests? - [x] Are tests passing? (`npm test` on both front/server) - [x] Is the linter passing? (`npm run eslint` on both front/server) - [x] Did you run prettier? (`npm run prettier` on both front/server) - [x] If you are adding a new features/services, did you run integration comparator? (`npm run compare-translations` on front) - [x] Did you test this pull request in real life? With real devices? If this development is a big feature or a new service, we recommend that you provide a Docker image to the community ([french forum](https://community.gladysassistant.com/)/[english forum](https://en-community.gladysassistant.com/)) for testing before merging. - [ ] If your changes modify the API (REST or Node.js), did you modify the API documentation? (Documentation is based on comments in code) - [ ] If you are adding a new features/services which needs explanation, did you modify the user documentation? See [the GitHub repo](https://github.com/GladysAssistant/v4-website) and the [website](https://gladysassistant.com). - [ ] Did you add fake requests data for the demo mode (`front/src/config/demo.js`) so that the demo website is working without a backend? (if needed) See [https://demo.gladysassistant.com](https://demo.gladysassistant.com). NOTE: these things are not required to open a PR and can be done afterwards / while the PR is open. ### Description of change Fix #2052

Specifically:

Default values will be selected for the ‹ scheduled trigger ›
A failing scene will no longer crash Gladys on startup

Topic		Replies	Views
Nouvelle vidéo : Gladys Assistant chez vous #5, Cyril et son café prêt tous les matins grâce à Gladys! Actualités	37	2492	October 10, 2022
Duplication de scènes [Archive] Demande de fonctionnalités feature-request-core , scene , feature-request-ux	40	2126	January 16, 2023
Pouvoir désactiver une scène sans la supprimer [Archive] Demande de fonctionnalités feature-request-core	10	1241	September 4, 2021
Utiliser la présence dans les scènes [Archive] Demande de fonctionnalités	21	2201	June 14, 2021
Gestion des fonctionnalités MQTT Configuration	10	956	October 13, 2021

Instance Gladys KO

Related topics