[Site Docusaurus Gladys] Build error since the latest Airtable additions

pierre-gilles · March 8, 2021, 5:41am

I have a small problem, do you remember you had developed the Airtable import script in English (which works great by the way!), but the issue is that since then, if I try to locally import the new devices added to Airtable, I can no longer launch Docusaurus ^^

I do:

node scripts/load_integrations.js

It adds 56 new files (images, json, etc..), and everything seems to work:

The problem is that when I restart the site, the « Integrations » page no longer works (it takes hours to load – unsuccessfully), and a build does not work.

I’m not sure it’s our fault, I have the impression that it may come from Docusaurus which struggles when we feed it too many images to process, and that we have exceeded a « limit ».

Or maybe, it’s our fault and there is a simple problem (like a missing image, something like that…) but I doubt it, it would give a clear error.

pierre-gilles · March 8, 2021, 6:59am

Hey @cicoub13 I fixed the issue by myself

Edit: no, the issue is not fixed. Infinite build on Github actions.

https://github.com/GladysAssistant/v4-website/pull/57

There was a double issue:

There was a hardcoded « en » folder in the integration view, and therefore some images were not found (Get new devices from AirTable by Pierre-Gilles · Pull Request #57 · GladysAssistant/v4-website · GitHub)
Some images that were on the repo and not on Airtable were in the « en » folder but not « fr », which broke the « fr » build

By fixing both, apparently I can display the site and the builds work!

pierre-gilles · March 8, 2021, 7:11am

I spoke too soon!

It’s still broken apparently, the build is infinite on GitHub Actions…

cf

https://github.com/GladysAssistant/v4-website/runs/2054279591?check_suite_focus=true

pierre-gilles · March 8, 2021, 7:18am

L’erreur sur Github actions:

2021-03-08T06:58:45.3525026Z ##[group]Run ./build.sh
2021-03-08T06:58:45.3525558Z e[36;1m./build.she[0m
2021-03-08T06:58:45.3571277Z shell: /bin/bash -e {0}
2021-03-08T06:58:45.3571855Z ##[endgroup]
2021-03-08T06:58:46.6430726Z 
2021-03-08T06:58:46.6433031Z > gladys-4-docusaurus@1.0.0 postinstall /home/runner/work/v4-website/v4-website
2021-03-08T06:58:46.6434443Z > cd en && npm install && cd ../fr && npm install
2021-03-08T06:58:46.6435062Z 
2021-03-08T06:58:56.4145957Z npm WARN @docusaurus/plugin-content-blog@2.0.0-alpha.58 requires a peer of @docusaurus/core@^2.0.0 but none is installed. You must install peer dependencies yourself.
2021-03-08T06:58:56.4235669Z npm WARN @docusaurus/plugin-content-docs@2.0.0-alpha.58 requires a peer of @docusaurus/core@^2.0.0 but none is installed. You must install peer dependencies yourself.
2021-03-08T06:58:56.4313076Z npm WARN @docusaurus/plugin-content-pages@2.0.0-alpha.58 requires a peer of @docusaurus/core@^2.0.0 but none is installed. You must install peer dependencies yourself.
2021-03-08T06:58:56.4391487Z npm WARN @docusaurus/plugin-debug@2.0.0-alpha.58 requires a peer of @docusaurus/core@^2.0.0 but none is installed. You must install peer dependencies yourself.
2021-03-08T06:58:56.4478723Z npm WARN @docusaurus/plugin-google-analytics@2.0.0-alpha.37 requires a peer of @docusaurus/core@^2.0.0 but none is installed. You must install peer dependencies yourself.
2021-03-08T06:58:56.4572167Z npm WARN @docusaurus/plugin-google-gtag@2.0.0-alpha.58 requires a peer of @docusaurus/core@^2.0.0 but none is installed. You must install peer dependencies yourself.
2021-03-08T06:58:56.4658446Z npm WARN @docusaurus/plugin-ideal-image@2.0.0-alpha.39 requires a peer of @docusaurus/core@^2.0.0 but none is installed. You must install peer dependencies yourself.
2021-03-08T06:58:56.4747398Z npm WARN @docusaurus/plugin-sitemap@2.0.0-alpha.58 requires a peer of @docusaurus/core@^2.0.0 but none is installed. You must install peer dependencies yourself.
2021-03-08T06:58:56.4829614Z npm WARN @docusaurus/preset-classic@2.0.0-alpha.58 requires a peer of @docusaurus/core@^2.0.0 but none is installed. You must install peer dependencies yourself.
2021-03-08T06:58:56.4909901Z npm WARN @docusaurus/plugin-google-analytics@2.0.0-alpha.58 requires a peer of @docusaurus/core@^2.0.0 but none is installed. You must install peer dependencies yourself.
2021-03-08T06:58:56.4991920Z npm WARN @docusaurus/plugin-sitemap@2.0.0-alpha.58 requires a peer of @docusaurus/core@^2.0.0 but none is installed. You must install peer dependencies yourself.
2021-03-08T06:58:56.5076229Z npm WARN @docusaurus/theme-classic@2.0.0-alpha.58 requires a peer of @docusaurus/core@^2.0.0 but none is installed. You must install peer dependencies yourself.
2021-03-08T06:58:56.5151365Z npm WARN @docusaurus/theme-search-algolia@2.0.0-alpha.58 requires a peer of @docusaurus/core@^2.0.0 but none is installed. You must install peer dependencies yourself.
2021-03-08T06:58:56.5249191Z npm WARN @endiliey/lqip-loader@3.0.2 requires a peer of file-loader@* but none is installed. You must install peer dependencies yourself.
2021-03-08T06:58:56.5371780Z npm WARN optional SKIPPING OPTIONAL DEPENDENCY: fsevents@2.1.3 (node_modules/fsevents):
2021-03-08T06:58:56.5376096Z npm WARN notsup SKIPPING OPTIONAL DEPENDENCY: Unsupported platform for fsevents@2.1.3: wanted {"os":"darwin","arch":"any"} (current: {"os":"linux","arch":"x64"})
2021-03-08T06:58:56.5470535Z npm WARN optional SKIPPING OPTIONAL DEPENDENCY: fsevents@1.2.13 (node_modules/watchpack-chokidar2/node_modules/fsevents):
2021-03-08T06:58:56.5474196Z npm WARN notsup SKIPPING OPTIONAL DEPENDENCY: Unsupported platform for fsevents@1.2.13: wanted {"os":"darwin","arch":"any"} (current: {"os":"linux","arch":"x64"})
2021-03-08T06:58:56.5610043Z npm WARN optional SKIPPING OPTIONAL DEPENDENCY: fsevents@1.2.13 (node_modules/webpack-dev-server/node_modules/fsevents):
2021-03-08T06:58:56.5612373Z npm WARN notsup SKIPPING OPTIONAL DEPENDENCY: Unsupported platform for fsevents@1.2.13: wanted {"os":"darwin","arch":"any"} (current: {"os":"linux","arch":"x64"})
2021-03-08T06:58:56.5613788Z 
2021-03-08T06:58:56.5639583Z audited 1716 packages in 9.485s
2021-03-08T06:58:57.5825084Z 
2021-03-08T06:58:57.5826242Z 124 packages are looking for funding
2021-03-08T06:58:57.5826956Z   run `npm fund` for details
2021-03-08T06:58:57.5827422Z 
2021-03-08T06:58:57.5838635Z found 933 vulnerabilities (927 low, 1 moderate, 5 high)
2021-03-08T06:58:57.5839828Z   run `npm audit fix` to fix them, or `npm audit` for details
2021-03-08T06:59:07.9415998Z npm WARN @docusaurus/plugin-content-blog@2.0.0-alpha.58 requires a peer of @docusaurus/core@^2.0.0 but none is installed. You must install peer dependencies yourself.
2021-03-08T06:59:07.9500549Z npm WARN @docusaurus/plugin-content-docs@2.0.0-alpha.58 requires a peer of @docusaurus/core@^2.0.0 but none is installed. You must install peer dependencies yourself.
2021-03-08T06:59:07.9581055Z npm WARN @docusaurus/plugin-content-pages@2.0.0-alpha.58 requires a peer of @docusaurus/core@^2.0.0 but none is installed. You must install peer dependencies yourself.
2021-03-08T06:59:07.9656513Z npm WARN @docusaurus/plugin-debug@2.0.0-alpha.58 requires a peer of @docusaurus/core@^2.0.0 but none is installed. You must install peer dependencies yourself.
2021-03-08T06:59:07.9729041Z npm WARN @docusaurus/plugin-google-analytics@2.0.0-alpha.37 requires a peer of @docusaurus/core@^2.0.0 but none is installed. You must install peer dependencies yourself.
2021-03-08T06:59:07.9812613Z npm WARN @docusaurus/plugin-google-gtag@2.0.0-alpha.58 requires a peer of @docusaurus/core@^2.0.0 but none is installed. You must install peer dependencies yourself.
2021-03-08T06:59:07.9891795Z npm WARN @docusaurus/plugin-ideal-image@2.0.0-alpha.39 requires a peer of @docusaurus/core@^2.0.0 but none is installed. You must install peer dependencies yourself.
2021-03-08T06:59:07.9969374Z npm WARN @docusaurus/plugin-sitemap@2.0.0-alpha.58 requires a peer of @docusaurus/core@^2.0.0 but none is installed. You must install peer dependencies yourself.
2021-03-08T06:59:08.0048846Z npm WARN @docusaurus/preset-classic@2.0.0-alpha.58 requires a peer of @docusaurus/core@^2.0.0 but none is installed. You must install peer dependencies yourself.
2021-03-08T06:59:08.0158446Z npm WARN @docusaurus/plugin-google-analytics@2.0.0-alpha.58 requires a peer of @docusaurus/core@^2.0.0 but none is installed. You must install peer dependencies yourself.
2021-03-08T06:59:08.0253014Z npm WARN @docusaurus/plugin-sitemap@2.0.0-alpha.58 requires a peer of @docusaurus/core@^2.0.0 but none is installed. You must install peer dependencies yourself.
2021-03-08T06:59:08.0331413Z npm WARN @docusaurus/theme-classic@2.0.0-alpha.58 requires a peer of @docusaurus/core@^2.0.0 but none is installed. You must install peer dependencies yourself.
2021-03-08T06:59:08.0408575Z npm WARN @docusaurus/theme-search-algolia@2.0.0-alpha.58 requires a peer of @docusaurus/core@^2.0.0 but none is installed. You must install peer dependencies yourself.
2021-03-08T06:59:08.0484329Z npm WARN @endiliey/lqip-loader@3.0.2 requires a peer of file-loader@* but none is installed. You must install peer dependencies yourself.
2021-03-08T06:59:08.0617168Z npm WARN optional SKIPPING OPTIONAL DEPENDENCY: fsevents@2.1.3 (node_modules/fsevents):
2021-03-08T06:59:08.0618792Z npm WARN notsup SKIPPING OPTIONAL DEPENDENCY: Unsupported platform for fsevents@2.1.3: wanted {"os":"darwin","arch":"any"} (current: {"os":"linux","arch":"x64"})
2021-03-08T06:59:08.0708620Z npm WARN optional SKIPPING OPTIONAL DEPENDENCY: fsevents@1.2.13 (node_modules/watchpack-chokidar2/node_modules/fsevents):
2021-03-08T06:59:08.0710347Z npm WARN notsup SKIPPING OPTIONAL DEPENDENCY: Unsupported platform for fsevents@1.2.13: wanted {"os":"darwin","arch":"any"} (current: {"os":"linux","arch":"x64"})
2021-03-08T06:59:08.0796188Z npm WARN optional SKIPPING OPTIONAL DEPENDENCY: fsevents@1.2.13 (node_modules/webpack-dev-server/node_modules/fsevents):
2021-03-08T06:59:08.0798586Z npm WARN notsup SKIPPING OPTIONAL DEPENDENCY: Unsupported platform for fsevents@1.2.13: wanted {"os":"darwin","arch":"any"} (current: {"os":"linux","arch":"x64"})
2021-03-08T06:59:08.0799947Z 
2021-03-08T06:59:08.0832702Z audited 1716 packages in 9.472s
2021-03-08T06:59:09.1281464Z 
2021-03-08T06:59:09.1282943Z 124 packages are looking for funding
2021-03-08T06:59:09.1284002Z   run `npm fund` for details
2021-03-08T06:59:09.1284496Z 
2021-03-08T06:59:09.1289505Z found 933 vulnerabilities (927 low, 1 moderate, 5 high)
2021-03-08T06:59:09.1290616Z   run `npm audit fix` to fix them, or `npm audit` for details
2021-03-08T06:59:09.3685909Z audited 152 packages in 23.574s
2021-03-08T06:59:09.4326662Z 
2021-03-08T06:59:09.4327502Z 4 packages are looking for funding
2021-03-08T06:59:09.4328030Z   run `npm fund` for details
2021-03-08T06:59:09.4328327Z 
2021-03-08T06:59:09.4337051Z found 1 high severity vulnerability
2021-03-08T06:59:09.4337897Z   run `npm audit fix` to fix them, or `npm audit` for details
2021-03-08T06:59:09.9692267Z 
2021-03-08T06:59:09.9694460Z > my-website@0.0.0 build /home/runner/work/v4-website/v4-website/en
2021-03-08T06:59:09.9695570Z > docusaurus build
2021-03-08T06:59:09.9696141Z 
2021-03-08T06:59:11.3906591Z Creating an optimized production build...
2021-03-08T06:59:11.8991700Z [info] [webpackbar] Compiling Client
2021-03-08T06:59:11.9217023Z [info] [webpackbar] Compiling Server
2021-03-08T07:13:44.4220901Z 
2021-03-08T07:13:44.4222642Z <--- Last few GCs --->
2021-03-08T07:13:44.4223196Z 
2021-03-08T07:13:44.4224703Z [3229:0x4ce0f70]   872349 ms: Mark-sweep 1990.9 (2067.7) -> 1973.9 (2067.0) MB, 2169.4 / 0.2 ms  (average mu = 0.244, current mu = 0.256) allocation failure scavenge might not succeed
2021-03-08T07:13:44.4227033Z [3229:0x4ce0f70]   872444 ms: Scavenge 1989.0 (2067.0) -> 1976.3 (2067.0) MB, 12.7 / 0.0 ms  (average mu = 0.244, current mu = 0.256) allocation failure 
2021-03-08T07:13:44.4229622Z [3229:0x4ce0f70]   872547 ms: Scavenge 1990.1 (2067.0) -> 1978.9 (2067.0) MB, 22.7 / 0.0 ms  (average mu = 0.244, current mu = 0.256) allocation failure 
2021-03-08T07:13:44.4230693Z 
2021-03-08T07:13:44.4232342Z 
2021-03-08T07:13:44.4233332Z <--- JS stacktrace --->
2021-03-08T07:13:44.4233856Z 
2021-03-08T07:13:44.4234496Z ==== JS stack trace =========================================
2021-03-08T07:13:44.4234969Z 
2021-03-08T07:13:44.4235555Z     0: ExitFrame [pc: 0x140a7f9]
2021-03-08T07:13:44.4236321Z Security context: 0x32e4e90008d1 <JSObject>
2021-03-08T07:13:44.4238348Z     1: /* anonymous */ [0x26edba36dd71] [/home/runner/work/v4-website/v4-website/en/node_modules/acorn/dist/acorn.js:~808] [pc=0x12746eec88ce](this=0x22c8d4304271 <Parser map = 0x1a1ae9a7ec39>,0x0a9642f8b859 <String[#3]: for>,0x2ee3d23404b1 <undefined>,0x2ee3d23404b1 <undefined>)
2021-03-08T07:13:44.4240491Z     2: /* anonymous */ [0x26edba36e1f1] [/home/runner/work/v4-website/v4-website/en/...
2021-03-08T07:13:44.4241521Z 
2021-03-08T07:13:44.4244430Z FATAL ERROR: Ineffective mark-compacts near heap limit Allocation failed - JavaScript heap out of memory
2021-03-08T07:13:44.4931042Z  1: 0xa18150 node::Abort() [node]
2021-03-08T07:13:44.4931907Z  2: 0xa1855c node::OnFatalError(char const*, char const*) [node]
2021-03-08T07:13:44.4932979Z  3: 0xb9705e v8::Utils::ReportOOMFailure(v8::internal::Isolate*, char const*, bool) [node]
2021-03-08T07:13:44.4934117Z  4: 0xb973d9 v8::internal::V8::FatalProcessOutOfMemory(v8::internal::Isolate*, char const*, bool) [node]
2021-03-08T07:13:44.4935284Z  5: 0xd54655  [node]
2021-03-08T07:13:44.4936094Z  6: 0xd54ce6 v8::internal::Heap::RecomputeLimits(v8::internal::GarbageCollector) [node]
2021-03-08T07:13:44.4937444Z  7: 0xd615a5 v8::internal::Heap::PerformGarbageCollection(v8::internal::GarbageCollector, v8::GCCallbackFlags) [node]
2021-03-08T07:13:44.4939291Z  8: 0xd62455 v8::internal::Heap::CollectGarbage(v8::internal::AllocationSpace, v8::internal::GarbageCollectionReason, v8::GCCallbackFlags) [node]
2021-03-08T07:13:44.4941620Z  9: 0xd64f0c v8::internal::Heap::AllocateRawWithRetryOrFail(int, v8::internal::AllocationType, v8::internal::AllocationOrigin, v8::internal::AllocationAlignment) [node]
2021-03-08T07:13:44.4944822Z 10: 0xd2b92b v8::internal::Factory::NewFillerObject(int, bool, v8::internal::AllocationType, v8::internal::AllocationOrigin) [node]
2021-03-08T07:13:44.4948355Z 11: 0x106defe v8::internal::Runtime_AllocateInYoungGeneration(int, unsigned long*, v8::internal::Isolate*) [node]
2021-03-08T07:13:44.4958799Z 12: 0x140a7f9  [node]
2021-03-08T07:15:45.0450697Z ##[error]The operation was canceled.

Ok JavaScript heap out of memory

pierre-gilles · March 8, 2021, 7:41am

Well, apparently the memory error is more of a consequence than a symptom of the problem…

The issue seems to be related to sharp, which struggles when there are too many images:

11:12:13 PM: (sharp:1883): GLib-GObject-WARNING **: 15:12:13.499: gtype.c:4265: type id '0' is invalid
11:12:13 PM: (sharp:1883): GLib-GObject-WARNING **: 15:12:13.499: can't peek value table for type '<invalid>' which is not currently referenced
11:12:13 PM: (sharp:1883): GLib-GObject-WARNING **: 15:12:13.499: gvalue.c:188: cannot initialize GValue with type '(null)', this type has no GTypeValueTable implementation

This leads to an infinite loop, then memory leak, then OOM JS.

If anyone has an idea, @VonOx or @cicoub13, I’m all ears

I’ll keep investigating in the meantime, but it’s really mysterious…

Edit: we’re not alone, the problem also exists with Gastby:
https://github.com/gatsbyjs/gatsby/issues/6291

pierre-gilles · March 8, 2021, 7:48am

Ok I have a lead.

Apparently there is a problem with versions below 0.27.1 of sharp.

Currently, with the version of Docusaurus we are using, we are on 0.22.1, hence the problem.

VonOx · March 8, 2021, 8:12am

Oh, I see! Sharp!

A few weeks ago, @Terdious and @damalgos added it to the Netatmo service, but it’s impossible to build ARM images with this package.

pierre-gilles · March 8, 2021, 8:16am

Well, I’m in the process of updating Docusaurus to its latest version, it’s not easy there are a lot of breaking changes ^^

VonOx · March 8, 2021, 8:21am

I just came across this, knowing that the action is executed on Ubuntu, it’s worth a try

https://github.com/lovell/sharp/issues/955#issuecomment-475532037

https://github.com/lovell/sharp/issues/2607#issuecomment-790931380

pierre-gilles · March 8, 2021, 8:25am

I don’t think we have anything to do at the system level

There is clearly a bug in the version of sharp we are using, and the real remedy is to update sharp!

VonOx · March 8, 2021, 8:29am

I agree, I’m just trying to unlock the thing

By the way, I just discovered the yarn function (no equivalent in npm as far as I know) to manage a version on a dependency

"resolutions": {
    "sharp": "^0.27.1"
  }

pierre-gilles · March 8, 2021, 9:49am

Well, I’ve been trying to update Docusaurus for several hours, and I’ve come across some pretty deep Docusaurus bugs

They have a problem with their Joi validation that expects « prependBaseUrlToHref » to be a string, while in their documentation they say it’s a boolean (which would make sense)…

github.com/facebook/docusaurus

packages/docusaurus-theme-classic/src/validateThemeConfig.js

333545aa9


      
          };
          exports.DEFAULT_CONFIG = DEFAULT_CONFIG;
          
          const NavbarItemPosition = Joi.string().equal('left', 'right').default('left');
          
          const BaseNavbarItemSchema = Joi.object({
            to: Joi.string(),
            href: URISchema,
            label: Joi.string(),
            className: Joi.string(),
            prependBaseUrlToHref: Joi.string(),
          })
            // We allow any unknown attributes on the links
            // (users may need additional attributes like target, aria-role, data-customAttribute...)
            .unknown();
          
          // TODO we should probably create a custom navbar item type "dropdown"
          // having this recursive structure is bad because we only support 2 levels
          // + parent/child don't have exactly the same props
          const DefaultNavbarItemSchema = BaseNavbarItemSchema.append({
            items: Joi.array().optional().items(BaseNavbarItemSchema),

This smells like a PR on the Docusaurus side…

pierre-gilles · March 8, 2021, 10:03am

Issue created in Docusaurus: "prependBaseUrlToHref" is defined as a string in Joi validation, and it's supposed to be a boolean in the doc · Issue #4363 · facebook/docusaurus · GitHub

I won’t have time to make a PR for them today, for now I’ve created an issue and we’ll see what they say..

In the meantime, I think we’re stuck. We can’t move to the latest version ^^

pierre-gilles · March 18, 2021, 6:22am

For your information, the latest version of Docusaurus (alpha-72) handles internationalization!

Today I’m trying to upgrade to this version, it looks really cool

By the way, I realized that the lead maintainer of this project is French, and I’m in a group with him ^^

pierre-gilles · March 18, 2021, 7:47am

For those who want to follow, the PR is there:

https://github.com/GladysAssistant/v4-website/pull/60

The work is massive, I think it will take me some time.

cicoub13 · March 18, 2021, 9:28am

Great initiative. Couldn’t you do it in two steps? Update while keeping the en/fr versions, then enable internationalization?

Don’t have much time to help, but for testing, yes

pierre-gilles · March 18, 2021, 9:33am

There are still quite a few differences, as the API has changed a lot between the version we were using and version 72, since it’s still in alpha.

Anyway, I’m on it I created an empty Docusaurus project, and I gradually copied everything into it. I almost have the EN site functional now.

Next, I’ll need to generate the FR translation files and copy the translations from the current site. It’s a bit of tedious work, but it has to be done ^^

pierre-gilles · March 19, 2021, 5:08am

I spent the whole morning on it, the site finally builds!!

I had to translate all the parts of the documentation that were in French but not in English.

Honestly, this new version of Docusaurus is really cool, it detects broken links, images that point to nothing, it’s really well done. I found a lot of issues on the site thanks to that

pierre-gilles · March 19, 2021, 6:37am

Here’s a preview of the site for those who want to test it:

[EDIT: the site is live! → https://gladysassistant.com/ to test!]

In short:

Now, the root of the site is the English version of the site
In « /fr » there is the French version of the site
I had to transfer all the FR articles that were not on the English site. I translated some of them, but not all because it’s really time-consuming… ^^

Normally, in terms of documentation, there is complete parity between the FR and EN versions of the site (everything translated :p)

Let me know what you think!

pierre-gilles · March 19, 2021, 7:27am

Well, I’ve put the site online, it was already much better for me than before

Don’t hesitate if you see anything, I’ll keep looking on my side!

Topic		Replies	Views
Un nouveau site pour Gladys 4? Développement	30	2860	July 21, 2020
Gladys Assistant: Ecrivons plus de contenu ensemble! Développement	65	1050	May 17, 2024
Le chemin vers Gladys 4 RC Actualités	25	2358	December 1, 2020
Rendre les menus homogène Développement	34	659	January 24, 2024
Point PR janvier 2023 Développement	19	801	January 20, 2023

[Site Docusaurus Gladys] Build error since the latest Airtable additions

Related topics