Reducing the app buildling time with concatenateModules: false

[slorber]

Docusaurus uses Webpack/Rspack to bundle your app, which by default uses optimization.concatenateModules: true for production builds.

While working on Docusaurus v3.8 optimizations, we found out that this optimization:

• can be very expensive on very large sites (see benchmarks on a 11k docs website)
• only saves 3% of JS of the total JS assets size

If you have a very large site, and build time is a concern, you have the option to make builds faster, at the cost of slightly increased JS size.

A good idea could be to disable this optimization for local and deploy preview builds, but keep it only for production deployments.

Here's an example of how to do this when deploying on Vercel or Netlify:

// See https://vercel.com/docs/environment-variables/system-environment-variables
const isVercelProductionDeployment = process.env.VERCEL && process.env.VERCEL_ENV === "production"

// See https://docs.netlify.com/configure-builds/environment-variables/
const isNetlifyProductionDeployment = process.env.NETLIFY && process.env.CONTEXT === "production";

const isProductionDeployment = isVercelProductionDeployment || isNetlifyProductionDeployment; 

// Site config
const siteConfig = {
  // ... other site config
  plugins: [
    function disableExpensiveBundlerOptimizationPlugin() {
      return {
        name: "disable-expensive-bundler-optimizations",
        configureWebpack(_config, isServer) {
          return {
            optimization: {
              concatenateModules: isProductionDeployment ? !isServer : false,
            },
          };
        },
      };
    },
  ],
};

---

Why not apply this by default?

By default, we believe it's better to favor UX (saving 3% JS for your site's visitors) rather than DX (reducing build times).

If you have a different opinion, it should be your responsibility to configure your optimizations accordingly. A good compromise is to only optimize for production deployment. Unfortunately, Docusaurus can't know if your docusaurus build is run for a test / deploy preview environment, or your real production deployment.

Also, the Rspack team is aware of a possible bottleneck on this optimization that doesn't scale well when the number of docs increases. They will likely improve things in the future, making this trick less impactful. We could make this easier to configure, but we also don't want to introduce a first-party config option that we'll likely remove later.

---

Share your results

Please share below how much this tip improved your build time; this may help other users decide if they should try it too!

You can use DOCUSAURUS_PERF_LOGGER=true to print additional logs and know how much time it takes to bundle your app.

I'll start with these results from a very large 11k docs website:

Before:

• Cold build, optimization enabled: 660s
• Warn build, optimization enabled: 550s

After:

• Cold build, optimization disabled: 170s 🔥
• Warn build, optimization disabled: 33s 🔥

[felipecrs]

@slorber I believe you can consider disabling these optimizations by default when not building for production. There's no risk in that, is there? A Docusaurus dev build is already very different than a production build.

PS: toggling this option made no difference in my build time, which remains around 20s after all 3.8 enhancements.

[slorber]

@slorber I believe you can consider disabling these optimizations by default when not building for production. There's no risk in that, is there? A Docusaurus dev build is already very different than a production build.

I'll have to check, but I believe we use mode: 'development' for docusaurus start and docusaurus build --dev, so the optimization is likely off in these cases. Eventually, I'll move the explicit config to the server config only so that it never affects client configs (including docusaurus start, but on the large site above I didn't really see a bottleneck so I'm not sure it's used by the dev server anyway.

PS: toggling this option made no difference in my build time, which remains around 20s after all 3.8 enhancements.

This option is unlikely to have a significant impact on small sites, because the problem is related to algorithmic complexity, and becomes more significant as the number of pages/js modules increases.

The site above is very large (11k docs), and the algorithm doesn't scale well as the number of pages increases. If your site builds in 20s, it's probably not large enough to be impacted by this perf bottleneck.

On our website (~1200 pages) and ReactNative.dev (~1950 pages,) we still see a very visible improvement.

See React Native website PR and the impact of this optimization removal here: facebook/react-native-website#4607

Default:

• Cold build: 37s
• Warm rebuild: 17s

With the expensive optimization disabled:

• Cold build: 19s
• Warm rebuild: 3s

[felipecrs]

Wow. This difference in the React Native website is crazy big. Definitely something worth trying, and to have in mind. Thank you!

[omonk]

Glad I came across this thread this morning! I had a read of the referenced thread where a site with 11k pages was taking a long time to build. We're experiencing similar issues but with under slightly different CI conditions.

We have a few hundred hand written documents in MD/MDX but also use the openapi plugin mentioned to generate more documents, in total 5980

fd -e md -e mdx -I | wc -l
> 5980

We are using vercel to deploy which means our build containers are limited without upgrading from Pro to use more compute. After encountering scaling issues in our production builds in vercel after adding more versions of our openapi spec we refined the experimental_faster option to this:

future: {
  experimental_faster: {
    swcJsLoader: false,
    swcJsMinimizer: false,
    swcHtmlMinimizer: false,
    lightningCssMinimizer: false,
    rspackBundler: true,
    mdxCrossCompilerCache: false,
  },
},

This is paired with the some memory tweaks for the node process

NODE_OPTIONS=--max-old-space-size=7168 yarn run build

The resulting build time is around 10 - 12mins.

I think given the details listed in #11140 our build times are actually pretty good? Either way, I am conscious of the fact that this nonstandard approach might come back and bite us in the future when v4 is released and we're stuck in 3.8.0

New conncatenateModules: false suggestion

I've stripped out the custom experiemental_faster config and replaced with the standard setup plus added the webpack configuration to turn off concatenateModules

    v4: true,
    experimental_faster: true,

Locally on an M1 Max this runs in around 2mins,

In Vercel unfortunately we have mixed results

First run, 20mins to OOM
Second run, 5mins to OOM
Third run, >23mins, no OOM but I cancelled the build

I will continue to play around with the experiemental_faster configs but thought it would be helpful to share my findings

[slorber]

It's unexpected to OOM on Vercel if it works locally. Make sure you are using the same memory settings everywhere, and it can also be helpful to share the errors you get. Also helpful to know which step fails (use DOCUSAURUS_PERF_LOGGER=true and show us the full build log including the error stacktrace).

It's possible that too much memory is consumed by the SSG workers. I believe there may be a bug in Node.js that prevents correct memory control over such threads, and there may not have enough memory on your Vercel builder to support the default number of threads.

As detailed in #10826 there's a secret env variable you can use to reduce the number of threads. I would try with DOCUSAURUS_SSG_WORKER_THREAD_COUNT=1 or DOCUSAURUS_SSG_WORKER_THREAD_COUNT=1 to see if it's the problem.

Note that you probably don't need --max-old-space-size=7168, that's likely too much for the size of your site. I think there's a bug in Node.js worker threads' memory limits that always allocate that much memory to all worker threads despite using the dedicated memory control options, so if you have 2 worker threads + the main process it may take up to 21gb of memory. I'm planning to investigate that later, but try to reduce that memory limit locally, you'll figure out that you probably can build your site locally with much less memory, and it may even fix the Vercel build problem too.