chadkillingsworth / closure-calculate-chunks Goto Github PK

This is transferred from google/closure-compiler#3931

Closure Compiler introduces closure-calculate-chunks as a tool to create an option to use chunks.

https://github.com/google/closure-compiler/wiki/Chunk-output-for-dynamic-loading#asking-closure-compiler-to-produce-chunks---the-easy-way

But it doesn't work well when we have a dependency referenced by goog.requireType because google-closure-deps doesn't list files referenced by goog.requireType as dependencies. closure-calculate-chunks accepts the deps.js generated by google-closure-deps so it means that closure-calculate-chunks cannot recognize the file because the file isn't in the deps.js.

• myapp.js

goog.provide("myapp");

// mylib isn't listed as a dependency of `myapp.js`
goog.requireType("mylib");

myapp.foo = a => console.log(a);

// goog.addDependency('../../../../../src/myapp.js', ['myapp'], [], {'lang': 'es6'});

I think this is caused by the following problems

• google-closure-deps doesn't list files referenced by goog.requireType (this is not a problem of Closure Compiler itself)
• There is no way to specify files only referenced by goog.requireType as inputs of chunks
  • We could pass the files as --js options unless the dependencies don't include any circular dependency.

If Closure Compiler won't support the case of a circular dependency caused by dependencies of goog.requireType, this seems to be solved only by fixing google-closure-deps.

Background: Our project currently uses closure-calculate-chunks to chunk the output into the main chunk ("blockly") and several other chunks ("blocks", and "javascript", "python", "php", etc., the latter collectively known as the "generators").

At the moment the second and subsequent chunks depend only on the first chunk, but I experimentally added an import statement that caused the "javascript" chunk to depend on the "blocks" chunk (specifically, by importing the entry point of the "blocks" chunk) as well as the "blockly" chunk.

After updating our deps.js using closure-make-deps, running our usual c-c-c command:

closure-calculate-chunks --closure-library-base-js-path ./closure/goog/base_minimal.js --deps-file ./tests/deps.js \
    --entrypoint 'core/blockly.js' --entrypoint 'blocks/blocks.js' --entrypoint 'generators/javascript/all.js' \
    --entrypoint 'generators/python/all.js'  # ... etc.

resulted in the error message:

Error: Invalid chunk definitions:
Chunk entrypoint blocks/blocks.js not found in chunk sources. Ensure that all imports of blocks/blocks.js are dynamic.

This was pretty confusing, because it appears to be factually incorrect: blocks/blocks.js appears both in deps.js and as an --entrypoint. After adding some console.log messages to lib/chunk-graph.js I was able to figure out that the problem was that all of the files for the "blocks" chunk were being hoisted into the "blockly" chunk, leaving no files left.

I was aware of c-c-c's hoisting ability, but had (perhaps naïvely!) expected it to detect that the "javascript" chunk depended on the "blocks" chunk.

Now I am trying to figure out how to get closure-calculate-deps to deal with inter-chunk dependencies not involving the first chunk. After reading the documentation, I noted that the --manual-entrypoint switch might be relevant; I tried:

closure-calculate-chunks --closure-library-base-js-path ./closure/goog/base_minimal.js --deps-file ./tests/deps.js  \
    --entrypoint 'core/blockly.js' --entrypoint 'blocks/blocks.js' --manual-entrypoint 'blocks/blocks.js:generators/javascript/all.js' \
    --entrypoint 'generators/python/all.js'  # ... etc.

And this did seem to have the desired effect. However:

• The documentation for --manual-entrypoint just says "Add a custom entrypoint for code that is not discoverable", which wasn't too helpful, because:
  • this doesn't exactly stand out as being the way to specify dependency, and
  • it's not clear how one would specify dependency upon more than one previous chunk—e.g., a diamond dependency, where b and c depend on a, and then d depends upon b and c, and [edit: I see that Closure Compiler doesn't actually support this.]
  • I wasn't aware that any entrypoint could be "discoverable" (though I guess this might have something to do with --package-json-entry-names, which I also don't understand).
• This caused c-c-c to reorder the chunks in its output, compared to its input.
  • Since all of our generators have an entrypoint named generators/<lang>/all.js, c-c-c ends up naming them all, all1, all2, etc., which is not very helpful.
  • I had therefore written some code to rename the chunks generated by c-c-c to correspond to our desired chunk names before passing them to Closure Compiler.
  • Previously I relied on them coming out in the same order they went in, but now I'm not sure what to do; it looks like I will have to look through the contents of each chunk and try to match the (now absolute!) pathnames against one of the entrypoints.
    • I even tried --naming-style=numbered in the hopes that they might come out in the order 0, 3, 4, 5, 1, 2 (and I could just index my original array of chunk entrypoints) but alas no.

I feel like I'm fighting the tool.

Can the documentation and/or code be improved to make it easier to do this (or at least easier to understand what one is doing wrong)?

Projects using native ECMAScript modules could use a command line flag to limit closure-calculate-chunks to only following ESM imports (import, import(), and export * from), and to ignore require(), goog.require(), and the like.

Code that uses the imperative RequireJS API for loading external libraries at runtime can cause tracebacks when closure-calculate-chunks attempts to parse them.

When run under node.js v12 (which is still in LTS), closure-calculate-chunks fails with the following error:

Error [ERR_MODULE_NOT_FOUND]: Cannot find package 'fs' imported from […]/node_modules/closure-calculate-chunks/lib/chunk-graph.js
    at packageResolve (internal/modules/esm/resolve.js:664:9)
    at moduleResolve (internal/modules/esm/resolve.js:705:18)
    at Loader.defaultResolve [as _resolve] (internal/modules/esm/resolve.js:798:11)
    at Loader.resolve (internal/modules/esm/loader.js:100:40)
    at Loader.getModuleJob (internal/modules/esm/loader.js:246:28)
    at ModuleWrap.<anonymous> (internal/modules/esm/module_job.js:47:40)
    at link (internal/modules/esm/module_job.js:46:36) {
  code: 'ERR_MODULE_NOT_FOUND'
}

This is obviously inconvenient for those of us who use it as part of our build pipeline, and who test building on older versions of node.js.