Gladys Assistant v4.8.6, with a new Gladys Plus backup algorithm!

News
1

Hello everyone!

I’m happy to launch today Gladys Assistant v4.8.6, an update dedicated to Gladys Plus backups (Gladys Plus | Gladys Assistant) :slight_smile: ( Still €59.99 per year, don’t hesitate!! )

Since it’s an exciting topic, I wanted to write a slightly more technical post to explain how backups now work.

To summarize in 4 points:

  • Backups are now performed between midnight and 2 AM local time of your instance; the time is randomly chosen each day to avoid overloading the Gladys Plus API, because you are now 60 uploading your backups each day, and I expect this number to increase significantly in the coming months.
  • Backups are uploaded in chunks (small 20MB pieces), rather than in one single block, so as not to overload the RAM of your instances and to be able to retry each small piece in case of failure.
  • Backups are uploaded directly via the object storage API of my hosting provider, and no longer proxied through the Gladys Plus backend. This helps relieve the backend and frees up incoming bandwidth.
  • It is now possible to see in Gladys the status of an ongoing and past backup, with a progress percentage, and in case of failure, the error message.

Screenshot 2022-05-09 at 17.37.27
Screenshot 2022-05-09 at 17.37.272468×1318 309 KB

Now, 2 nice diagrams :smiley:

At the server architecture level, it now looks like this:

process upload gladys plus@2x
process upload gladys plus@2x1756×964 41 KB

The upload is done via signed URLs, and no longer via backend proxying as it was before.

At the upload process level, I made a diagram to illustrate the new upload algorithm:

Processus de sauvegarde Gladys Plus@2x
Processus de sauvegarde Gladys Plus@2x2458×3484 498 KB

I hope this is very clear — if you have any questions, feel free to put them in the comments :wink:

I wrote this very transparent post to also show the “behind the scenes” of a development, and that behind “minor” developments (all of this is about stability, there are no new features), there can be big considerations and time spent in terms of spec + development + testing.

Personally, I love these “infrastructure” developments which are very close to what I do for my freelance clients alongside Gladys :slight_smile:

The full CHANGELOG is available here.

How to update?

If you installed Gladys with the official Raspberry Pi OS image, your instances will update automatically in the coming hours. This can take up to 24 hours, don’t panic.

If you installed Gladys with Docker, make sure you are using Watchtower (See the documentation)

7 Likes
2

Needless to say, I don’t get a thing… But hats off to the work!! :ok_hand::ok_hand:

4 Likes
3

Great, thanks :slight_smile:
I was about to start writing « but since you’re doing all that in the backend, what’s the logic? », while I was reading the article. But you had the great idea of explaining it very clearly with diagrams.

Thanks for taking the time to document all of that, for greater transparency and more interest in following the development of Gladys!

2 Likes
4

For now the instances seem to be updating smoothly and I see backups going through, everything seems to be going well!

I just saw a failed backup for @Mastho — can you go check under « Settings », « Tasks » and see what went wrong? :slight_smile:

5

Hi @pierre-gilles,

Great job on this new backup process!

Here is the error:

Gladys Plus Backup
Gladys Plus Backup845×130 8.52 KB

6

Thanks @Mastho, I think I know what happened! :slight_smile:

I’ll fix that!

1 Like
7

I think I found it, I fixed it in a PR:

I’ll merge that as soon as possible and deploy a version :slight_smile:

1 Like
8

Well done on this new system, very nice to be able to follow the progress of the backup. Just one small thing to maybe revisit: the rounding of the % in the Tasks view, we regularly end up with xx.46666666667 etc. nothing serious!!^^

I thought you had updated the number of backups but apparently not, I still have the same number ^^

For info as well (I don’t think there’s much that can be done):
At the start of the backup, the database copy was correctly made in the Backup folder. In Tasks I therefore went to 10% quite quickly (about 2 minutes it seems) then stayed stuck at that stage for about 10 minutes. So I checked the logs and noticed that one of my cameras had been down for a few days, it was therefore producing poll errors every 10s. I then set this one to a poll of 1 minute (feature idea by the way: be able to disable a camera’s poll without having to delete it ^^) and the backup restarted … and finished quietly … would the error be heavier to handle than saving the image?? Surprising!! Or coincidence?

9

Indeed, I fixed that in the fix PR! :slight_smile:

In fact the first 3 steps (sqlite3 backup + gzip + encryption) are steps where I don’t have progress feedback, so they are 10% blocks in the backup progress.

0 → 10% = sqlite backup
10% → 20% => gzip
20% → 30% => openssl
30% → 100% => Upload

I know it’s frustrating when progress bars aren’t linear, but in this case I don’t really have a choice: I know when these commands start/end, but I have nothing in between :smiley:

Mmm, either coincidence, or a bug, or the websocket had dropped and then came back, which could give that impression in the interface that it had stopped and then resumed

1 Like
10

Regarding this topic, I just did it this morning :slight_smile:

1 Like
11

Gladys Assistant v4.8.7 has been deployed with fixes :white_check_mark:

5 Likes
12

Hello everyone!

I’m getting back to you 5 days after the deployment to show you the impact of this development on the bandwidth used on the Gladys Plus backend:

Screenshot 2022-05-20 at 15:57:31 fr
Screenshot 2022-05-20 at 15:57:31 fr2000×630 152 KB

The impact is clear: by stopping all uploads from going through the backend, bandwidth usage is decreasing, and above all there are no longer any bandwidth usage « spikes ».

On this 14-day graph it’s not very clear, but on a 1-hour graph we’re talking about network peaks of 150–200 Mbps easily, which could have impacted other users using Gladys Plus at the same time.

This development therefore brings long-term stability, and above all will allow Gladys Plus to welcome hundreds of new users without problems :wink:

6 Likes
13

Wait, I’m not there yet :smiley:

1 Like
14

:+1: Good job ..

1 Like
15

Look closely by zooming in, at each peak it says @Terdious @Terdious @Terdious :rofl::rofl:

3 Likes
16

I think @lmilcent has far surpassed me :sweat_smile:

Indeed @pierre-gilles the difference is striking!! Good news ^^

@VonOx, I always knew fiber was overrated :wink::rofl:

2 Likes
17

Never mind, I’m two releases behind, Alexa yelling at me because « there must be a problem with my router ».

Anyway, if all goes well tomorrow things will move fast

1 Like
18

Almost 5GB locally :sweat_smile:
But I need to clean all that up, I have quite a bit of data that’s not very useful.

2 Likes