Internal tracking: GITHUB-150

After bundling using bin/baler build getting TypeError: require is not a function error

attached error log for your reference.

• require_js error.log

I assume, that the problem has to do with insufficient memory on my vserver but i just wanted to ask if there is something i can fix to get baler running.

Running on Magento 2.3.4 and getting the following ERROR when trying to run baler.

[risingbytes@np500 public_html]$ baler baler ✔ Collected theme/module data 91ms /home/risingbytes/.nvm/versions/node/v13.7.0/bin/node[17854]: ../src/node_platform.cc:63:std::unique_ptr<long unsigned int> node::WorkerThreadsTaskRunner::DelayedTaskScheduler::Start(): Assertion (0) == (uv_thread_create(t.get(), start_thread, this))' failed.
node[17847]: pthread_create: Resource temporarily unavailable
1: 0xa9d570 node::Abort() [/home/risingbytes/.nvm/versions/node/v13.7.0/bin/node]
2: 0xa9d5f7 [/home/risingbytes/.nvm/versions/node/v13.7.0/bin/node]
3: 0xb082aa node::WorkerThreadsTaskRunner::WorkerThreadsTaskRunner(int) [/home/risingbytes/.nvm/versions/node/v13.7.0/bin/node]
/home/risingbytes/.nvm/versions/node/v13.7.0/bin/node[17865]: ../src/node_platform.cc:63:std::unique_ptr node::WorkerThreadsTaskRunner::DelayedTaskScheduler::Start(): Assertion `(0) == (uv_thread_create(t.get(), start_thread, this))' failed.
4: 0xb0836b node::NodePlatform::NodePlatform(int, node::tracing::TracingController*) [/home/risingbytes/.nvm/versions/node/v13.7.0/bin/node]
5: 0xa6e1a1 node::InitializeOncePerProcess(int, char**) [/home/risingbytes/.nvm/versions/node/v13.7.0/bin/node]
node[17853]: pthread_create: Resource temporarily unavailable
1: 0xa9d570 node::Abort() [/home/risingbytes/.nvm/versions/node/v13.7.0/bin/node]
6: 0xa6e6a3 node::Start(int, char**) [/home/risingbytes/.nvm/versions/node/v13.7.0/bin/node]
7: 0x7f3c7abe0545 __libc_start_main [/lib64/libc.so.6]
2: 0xa9d5f7 [/home/risingbytes/.nvm/versions/node/v13.7.0/bin/node]
8: 0xa0afc5 [/home/risingbytes/.nvm/versions/node/v13.7.0/bin/node]
3: 0xb082aa node::WorkerThreadsTaskRunner::WorkerThreadsTaskRunner(int) [/home/risingbytes/.nvm/versions/node/v13.7.0/bin/node]
4: 0xb0836b node::NodePlatform::NodePlatform(int, node::tracing::TracingController*) [/home/risingbytes/.nvm/versions/node/v13.7.0/bin/node]
5: 0xa6e1a1 node::InitializeOncePerProcess(int, char**) [/home/risingbytes/.nvm/versions/node/v13.7.0/bin/node]
6: 0xa6e6a3 node::Start(int, char**) [/home/risingbytes/.nvm/versions/node/v13.7.0/bin/node]
7: 0x7f5813e93545 __libc_start_main [/lib64/libc.so.6]
8: 0xa0afc5 [/home/risingbytes/.nvm/versions/node/v13.7.0/bin/node]

Error: spawn ENOMEM
at ChildProcess.spawn (internal/child_process.js:405:11)
at spawn (child_process.js:548:9)
at Object.fork (child_process.js:116:10)
at ChildProcessWorker.initialize (/home/risingbytes/baler/node_modules/jest-worker/build/workers/ChildProcessWorker.js:137:44)
at new ChildProcessWorker (/home/risingbytes/baler/node_modules/jest-worker/build/workers/ChildProcessWorker.js:127:10)
at WorkerPool.createWorker (/home/risingbytes/baler/node_modules/jest-worker/build/WorkerPool.js:44:12)
at new BaseWorkerPool (/home/risingbytes/baler/node_modules/jest-worker/build/base/BaseWorkerPool.js:82:27)
at new WorkerPool (/home/risingbytes/baler/node_modules/jest-worker/build/WorkerPool.js:30:1)
at new JestWorker (/home/risingbytes/baler/node_modules/jest-worker/build/index.js:131:26)
at Object.createMinifier (/home/risingbytes/baler/src/createMinifier.ts:9:21)
`

In Chrome, a preload tag ends up pushing the priority of a resource ahead of any other, non-preloaded resource. This is detrimental to the FMP time, because CSS ends up prioritized after the preloaded JS.

Note the special treatment of core-bundle.js, getting its first byte before any other asset

Note: The timeline above shows all resources sharing bandwidth, but devtools is lying. Each resource is getting downloaded exclusively

Asset order in the DOM

I verified through chrome://net-export/ that Chrome is signaling to the server that core-bundle.js should be the first resource sent after the initial HTML Document. Below are the streams for the first few critical assets, in the order the streams were created:

Looking through HTTP2_SESSION_RECV_DATA events in the net export log, you can see the real order the server flushed the responses in below. Note that Chrome used the exclusive bit on every stream, so assets are given 100% bandwidth and downloaded in the order specified between stream weight and priorities.

Real Download Order:

• Initial HTML Document
• core-bundle.js
• styles-l.css
• styles-m.css
• calendar.css
• require.js

core-bundle.js and calendar.css are both given a parent stream ID of 0. According to the H2 spec, when the exclusive bit is used and multiple dependencies are added to a parent stream, it should create this tree

parent stream > first dependency > second dependency

Because of this, core-bundle.js gets downloaded before any of the css on the page, even though the CSS has a higher stream weight. It seems like usage of the exclusive bit on every stream causes stream weights to be ignored, based on my reading of the spec. In particular, these 2 parts:

An exclusive flag allows for the insertion of a new level of
dependencies. The exclusive flag causes the stream to become the
sole dependency of its parent stream, causing other dependencies to
become dependent on the exclusive stream
https://tools.ietf.org/html/rfc7540#section-5.3.1

Streams with the same parent SHOULD be allocated resources
proportionally based on their weight
https://tools.ietf.org/html/rfc7540#section-5.3.2

Since the exclusive bit on every stream means no 2 resources will share a parent, the stream weight is never used.

IMO, Chrome's handling of H2 priorities here is wrong/buggy. If Chrome knows it's going to assign a higher weight to calendar.css than to core-bundle.js, it should be making calendar.css a dependency of parent 0 before core-bundle.js, which would result in the JS being prioritized immediately after the CSS, fixing the FMP timing.

baler has 2 pieces that are necessary for bundling to work within Magento.

• baler - an npm package used to package bundles and generate configuration
• Magento_Baler - a Magento module that injects necessary <link rel="preload" /> tags into the head of the document

Requirements

1. Module must be named Magento_Baler
2. Module must be published to Magento's composer repository
3. Module must be compatible with all versions of Magento supported by the Cloud team

Artifacts made available by baler

When baler is run, is spits out 2 different types of artifacts

1. core-bundle.js
2. baler-preload-map.json (see #3)

When should Magento_Baler be considered enabled for a given theme?

1. When Magento_Baler is an enabled module in the store
2. When the locale folder for the current theme contains a file called requirejs-baler-config.js

What Work Should Magento_Baler Do?

Loading of Core Bundle

On each request:

1. Replace requirejs-config.js in the head of the document with requirejs-baler-config.js
2. In the head of the document, add a preload tag that points to balerbundles/core-bundle.js in the root of the current theme's current locale

Loading of Dynamic Dependencies

On each request

1. Collect a list of file paths for all .phtml files being used to render the current request
2. Read/Parse baler-preload-map.json, which is a sibling of core-bundle.js in the root of the locale.
3. Create a new list, preloads
4. For each key in baler-preload-map.json:
   • Check if the .phtml file at that path was used to render
   • If yes, append the dependencies to preloads
5. De-duplicate the preloads list
6. For each entry in preloads, add a preload tag to the head of the document

Disabling Incompatible Features

The following features should be disabled when Magento_Baler is enabled:

• JS Bundling
• JS Minification

Stretch Goals

• Make sure the list of .phtml files loaded keeps the order these were loaded on the page. When injecting preload tags in the head, inject them in the same order that the .phtml files were rendered (this effectively prioritizes content higher in the document)

After setting up baler. I am experiencing the following errors

• MiniCart not loading [After adding product]
  

• Order Summary not loading in CartPage
  

• Checkout page not loading
  

We've currently hand-written all the code necessary to find all enabled modules/themes installed in a store.

This was done because we did not want to invoke bin/magento and incur the bootstrapping cost of Magento core every time baler is run.

However, @Vinai pointed out to me today that it's just as fast (if not slightly faster) to invoke autoload.php, and then dump values from the ComponentRegistrar:

php -r "require 'vendor/autoload.php'; echo json_encode([(new \\Magento\\Framework\\Component\\ComponentRegistrar)->getPaths('theme'),(new \\Magento\\Framework\\Component\\ComponentRegistrar)->getPaths('module')]);"

Switching to this would resolve several open bugs, and allow us to delete several hundred lines of code. In addition, the maintenance burden will be significantly smaller when Magento core makes any changes to the theme/module lookup or registration rules.

I have been struggling with getting baler working in my magento 2 store where i did run de build command but stays stuck at creating core bundle file. It has been running for 24 hours now. I have also followed multiple tutorials on how to install it so im sure where it goes wrong.

Screenshot: https://prnt.sc/rzdeo8

Hey! First of all I need to say that I'm really impressed with the performance of this solution an ease of use in relation to other tools which take advantage of headless browsers etc., great work!

Unfortunately, when I gave baler for a spin it seems like we hit the same obstacle that I did and documented under https://github.com/magesuite/magepack#modules-with-mixins-defined-cannot-be-included-in-a-bundle

The issue comes from mixins.js file and its processNames method which is (as far as I understand) responsible for prepending given modules' names with mixins! prefix in case certain module has any mixins defined for it so that they can be applied when it is loaded. The problem is that in order to get a path for said module, getPath is called, which then calls require.toUrl. Multiple calls later we land inside require's nameToUrl method:

nameToUrl: function (moduleName, ext, skipExt) {
                var paths, syms, i, parentModule, url,
                    parentPath, bundleId,
                    pkgMain = getOwn(config.pkgs, moduleName);

                if (pkgMain) {
                    moduleName = pkgMain;
                }

                bundleId = getOwn(bundlesMap, moduleName); // OH NO

                if (bundleId) { // NO NO NO PLEASE NO
                    return context.nameToUrl(bundleId, ext, skipExt);
                }

As you may have figured out, for each of our modules that were included in core bundle we are getting balerbundles/core-bundle instead of the actual name. Because of the fact that stock Magento makes use of mixins, even for providing jquery security patches I would say this issue is critical and should be addressed ASAP before anyone uses this solution in production.

There are 2 possible ways this can be solved:

1. Don't include modules which have any mixins defined in the bundle, which will hurt performance and I'm not sure if it is even possible to determine that 100% statically.
2. Make mixins.js bundling-aware by reimplementing whole toUrl to nameToUrl chain so that it doesn't have if (bundleId) { check in it.

That's all that I can see right now, I'll try to hack around a bit to see what the simplest route would be.

I have two store view with different languages: en_US and zh_Hant_HK, after I run "baler build" command, I can only find the bundle file in en_US folder. Seems the regex rule in src/magentoFS.ts not support the locale such as "zh_Hant_HK".

Hello,

I followed all the steps in https://github.com/magento/baler/blob/master/docs/ALPHA.md
And when I ran the command 'baler' everything was looking fine. For each Magento theme I got the command line message 'Created core bundle file' and 'Minified core bundle and RequireJS', however I'm not seeing any bundle in the project. Also checking the store url I get all JS files separate so no actual bundle was generated.

Am I missing something?

Kind regards,

EDIT: Found the bundle under balerbundles/core-bundle.js

The output from the CLI right now is not great.

• Noisy
• Incomplete
• There are some long tasks (minification) which take a long time, and it's not clear from the CLI

Developer experience is important for adoption, so I'd like to get this done ASAP.

TODO:

• Create a semi-generic type-safe task/event tracker that can be used for CLI output, but also other things (like the eventual --verbose flag and debug logs)

https://www.npmjs.com/package/update-notifier

After uninstalling the conflicting module for testing purposes (see issue #8) and then running baler again I got the following typeError:

TypeError: Cannot read property 'area' of undefined
    at Object.bundleThemes ([magento-root]/baler/src/index.ts:24:19)
    at Object.run ([magento-root]/baler/src/cli.ts:23:27)

This leads to the following code:

export async function bundleThemes(
    magentoRoot: string,
    store: StoreData,
): Promise<BundleResult[]> {
    const { components, deployedThemes } = store;

    const pendingBundleResults: ReturnType<typeof bundleSingleTheme>[] = [];
    for (const themeID of deployedThemes) {
        const theme = components.themes[themeID]; //<< LEADS TO HERE 2/2
        if (theme.area !== 'frontend' || theme.themeID === 'Magento/blank') {  //<< HERE 1/2
            continue;
        }

I'm not unsure of how to debug this but it seems strange that the theme val is undefined, unless the components.themes array doesn't match IDs with IDs in deployed Themes?

Also: I'm running this on a dev-server - should that even work, or do i need to test on production mode?

Edit by @DrewML: Bug has been fixed, but a more thorough solution should still be implemented. See later comments in thread for more details

Original Issue

In a multistore setup with different languages, some translations like for "Total" in checkout cart or the ones for the "add to cart" button, just after clicking it, fall back to the ones given for the default store view.

This only happens with the Baler module being enabled.

A colleague of mine debugged it to magento/module-translation/view/base/web/js/image-translation-dictionary.js

This was introduced in the following commit:
magento/magento2@71c781d
Which was seen as a resolution for
#13

• terser

Requirements

1. For each theme that is processed, all deployed .js files that are not in core-bundle.js should be minified through terser
2. Minified files should be written in-place (same file name) to avoid the need for any path rewriting at build or runtime.
3. Minification should happen in worker threads, to speed up compilation time and avoid blocking the main thread's bundling work

Pieces

• For worker-based minification, there's already the createMinifier module that will create a pool of minification workers. We'll want to add a minifyFromFile API, though, so we don't need to read loose files into memory in the main process and then serialize/transfer/parse in the worker

#43 (comment)

An example of an adhoc script can be found, but we should probably just use something off-the-shelf (suggestions in the linked comment)

Hi!

First of all, thank you working on Baler. I'm not using it in production yet but preliminary results are looking promising!

I've had it working with only minor hick-ups on 2.3.3 but this morning I tried to build 2.3.4 and got a crash in the frontend. It seems like Baler assumes that text!-dependencies are always HTML files and so things like 'text!js-translation.json' (introduced here) doesn't work.

I did the simplest fix I could think of just to fix my crash:

diff --git a/src/createBundleFromDeps.ts b/src/createBundleFromDeps.ts
index 897a047..3fab6f4 100644
--- a/src/createBundleFromDeps.ts
+++ b/src/createBundleFromDeps.ts
@@ -58,12 +58,13 @@ async function getFinalModuleSource(
     const path = join(baseDir, resolver(dep).modulePath);
     const source = await readFile(path, 'utf8');
     const isHTML = extname(path) === '.html';
+    const isJSON = extname(path) === '.json';
     const shims = getShimsForModule(dep, requireConfig);
     const hasDefine = isAMDWithDefine(source);
     const isNamed = isNamedAMD(source);
     const hasInvalidShim = hasDefine && !!shims;
 
-    if (isHTML) {
+    if (isHTML || isJSON) {
         return { dep, file: wrapTextModule(dep, source), hasInvalidShim };
     }

But I suppose a better long-term fix would be to look at the text!-prefix instead of the file extension. What do you think?

Cheers!

Hello,

can someone suggest where i can find initializeCLI
this file and dist directory inluded in barel

root@vmi223508:~/baler# tail -1 bin/baler
require('../dist/cli/initializeCLI').initializeCLI(process.argv, process.cwd());

but dist directory does not exist

root@vmi223508:~/baler# ls
bin docs jest.config.js node_modules package.json package-lock.json prettier.config.js prettier-globs README.md src tsconfig.json

due to this barel produce error below

root@vmi223508:/baler# node -v
v12.13.1
root@vmi223508:/baler# baler -v
Error: Cannot find module '../dist/cli/initializeCLI'
Require stack:

• /root/baler/bin/baler
  at Function.Module._resolveFilename (internal/modules/cjs/loader.js:797:15)
  at Function.Module._load (internal/modules/cjs/loader.js:690:27)
  at Module.require (internal/modules/cjs/loader.js:852:19)
  at require (internal/modules/cjs/helpers.js:74:18)
  at Object. (/root/baler/bin/baler:34:1)
  at Module._compile (internal/modules/cjs/loader.js:959:30)
  at Object.Module._extensions..js (internal/modules/cjs/loader.js:995:10)
  at Module.load (internal/modules/cjs/loader.js:815:32)
  at Function.Module._load (internal/modules/cjs/loader.js:727:14)
  at Function.Module.runMain (internal/modules/cjs/loader.js:1047:10)

I also try find similar file and directory in magento installed directory, but also without success

Can someone suggest, where I can find dist directory with related files

magento/magento2#23821

No work to be done on this project, just keeping this here for tracking/visibility

New line is missing after sourceMappingURL hint causing next merged file be on the same line as the comment:

// some code
gento_Theme/js/theme"]})}(require);
//# sourceMappingURL=requirejs-bundle-config.js.map;try { <==== this is a problem
    if (!window.localStorage || !window.sessionStorage) {
        throw new Error();
    }

    localStorage.setItem('storage_test', 1);
    localStorage.removeItem('storage_test');

Steps to reproduce:

• Install Baler, Magento Baler module;
• Fulfil Baler requirements: JS bundling, minification are disabled;
• Generate static and Baler bundles;
• Generated requirejs-bundle-config.js and balerbundles/core-bundle.js have no new line in the end
• Merged JS files are invalid.

core-bundle (copy).zip
requirejs-bundle-config.zip

Sorry I have one question, Magento2 is slow and what you made is really useful. We need every piece of benefit which can reduce the page load time.
And our customers are checking every minute the Google pagespeed score.
I could not find excatly when to run Baler?

In Developer mode or production mode?
I saw you instructed static content deploy, so it must run on production mode since there's no need for static content deploy in developer mode...

I made all the installation step by step (baler is working and telling it finished the bundling with success) but still Google pagespeed advice to use Baler instead of standart bundling of magento.
What I'm doing wrong I dont know

When to flush cache, or we dont need to flush cache, or cache flush destroys everything? which is true.

We are magento implementor company and all other developer, no one made successful baler installation until now...

Please give hand my client's site is https://www.stockperfume.com

Thanks in advance.,

I was able to apple baler on my local instance but I'm struggling to apply it to Magento Cloud. Is there any way to make baler command to be available in the Magento Cloud auto deployment process?

When I try to run baler, I get the following error:

internal/modules/cjs/loader.js:638
throw err;
^

Error: Cannot find module 'source-map-support/register'
at Function.Module._resolveFilename (internal/modules/cjs/loader.js:636:15)
at Function.Module._load (internal/modules/cjs/loader.js:562:25)
at Module.require (internal/modules/cjs/loader.js:692:17)
at require (internal/modules/cjs/helpers.js:25:18)
at Object. (/home/apache/.local/bin/baler:35:1)
at Module._compile (internal/modules/cjs/loader.js:778:30)
at Object.Module._extensions..js (internal/modules/cjs/loader.js:789:10)
at Module.load (internal/modules/cjs/loader.js:653:32)
at tryModuleLoad (internal/modules/cjs/loader.js:593:12)
at Function.Module._load (internal/modules/cjs/loader.js:585:3)

Hi,

I'm trying baler in a Docker container, and I have infinite spinner with the message (Magento/luma) Create core bundle file. It looks like the baler does not do anything, and I want to enable a verbose mode or tracing.

This can be done with --trace CLI argument; however, the filename is generated based on current datetime, and I'd like to be able to specify the trace file name. In my case it's /dev/stdout.

Please add a parameter with the trace filename.

These Mirasvit modules are really throwing some curve balls.

Error: Found > 1 "module.xml" for the module at path "vendor/mirasvit/module-search"
    at findRealModuleRoot (/Users/scottfriend/Sites/baler/src/magentoFS.ts:247:15)
    at processTicksAndRejections (internal/process/task_queues.js:89:5)
    at getModuleConfig (/Users/scottfriend/Sites/baler/src/magentoFS.ts:204:28)
    at async Promise.all (index 151)
    at getComposerComponents (/Users/scottfriend/Sites/baler/src/magentoFS.ts:332:18)
    at async Promise.all (index 0)
    at Object.getComponents (/Users/scottfriend/Sites/baler/src/magentoFS.ts:62:37)
    at async Promise.all (index 1)
    at Object.collectStoreData (/Users/scottfriend/Sites/baler/src/collectStoreData.ts:15:58)
    at Object.run (/Users/scottfriend/Sites/baler/src/cli.ts:20:19)

Here's the abbreviated directory structure of this module:
vendor/mirasvit/module-search

• CHANGELOG.md
• composer.json
• src
  • Search
    • ...
    • composer.json
    • etc
      • ...
      • module.xml
    • ...
    • registration.php
  • SearchElasticNative
    ...
    • composer.json
    • etc
      • ...
      • module.xml
    • ...
    • registration.php

I'm really not sure why it makes sense to have two Mage modules inside one composer module =/

When running baler build with our themes it seems to not complete the task and create any files and gives no output as to what could be going wrong, but seems to work fine for the Magento/luma theme:

Optimization is done, but stats have not been implemented in the CLI yet
- Collect theme/module data
✔ Collected theme/module data 250ms
- (Vendor/parenttheme) Create core bundle file
- (Vendor/childtheme) Create core bundle file
- (Magento/luma) Create core bundle file
✔ (Magento/luma) Created core bundle file 773ms
- (Magento/luma) Minify core bundle and RequireJS config
✔ (Magento/luma) Minified core bundle and RequireJS 10.5s

This is using the current master branch and v10.17.0 of node.

I got following Error when i visit my homepage

Uncaught TypeError: Cannot read property 'setConf' of undefined                       customer-data.js:33
    at customer-data.js:33
    at Object.execCb (require.js:1650)
    at Module.check (require.js:866)
    at Module.<anonymous> (require.js:1113)
    at require.js:132
    at require.js:1156
    at each (require.js:57)
    at Module.emit (require.js:1155)
    at Module.check (require.js:917)
    at Module.enable (require.js:1143)

Hey @DrewML

We're using Baler for projects, but since today we're unable to deploy because Baler gives an error. Please see error code below. We don't see any new commits in Baler and the commits that worked before in our projects are failing now as well (no code changes). We've been able to reproduce this in our development environment as well. Changing "default" to another name seems to resolve the issue.

baler stderr: /chroot/home/nextmp/.config/yarn/global/node_modules/@magento/baler/dist/createBundleFromDeps.js:14
    Object.defineProperty(o, "default", { enumerable: true, value: v });
           ^
TypeError: Cannot redefine property: default
    at Function.defineProperty (<anonymous>)
    at __setModuleDefault (/chroot/home/nextmp/.config/yarn/global/node_modules/@magento/baler/dist/createBundleFromDeps.js:14:12)
    at __importStar (/chroot/home/nextmp/.config/yarn/global/node_modules/@magento/baler/dist/createBundleFromDeps.js:22:5)
    at Object.<anonymous> (/chroot/home/nextmp/.config/yarn/global/node_modules/@magento/baler/src/createBundleFromDeps.ts:18:1)
    at Module._compile (internal/modules/cjs/loader.js:778:30)
    at Object.Module._extensions..js (internal/modules/cjs/loader.js:789:10)
    at Module.load (internal/modules/cjs/loader.js:653:32)
    at tryModuleLoad (internal/modules/cjs/loader.js:593:12)
    at Function.Module._load (internal/modules/cjs/loader.js:585:3)
    at Module.require (internal/modules/cjs/loader.js:692:17)
	from /var/lib/gems/2.5.0/gems/sshkit-1.21.0/lib/sshkit/runners/parallel.rb:11:in `block (2 levels) in execute'

That is a list of loaded JS files in waterfall order.

Pay attention to the order of files core-bundle.js and jquery.cookie.js
core-bundle.js includes jquery-storageapi.js at the same time, storageapi is depended from
'jquery/jquery-storageapi': {'deps': ['jquery/jquery.cookie']}

As you see from Image jquery.cookie.js is loaded later bundle. In result, I have some Js error because of undefined Jquery methods.

Looks like all files included in the bundle should be checked for dependencies, and I afraid they should be sorted in the bundle in the correct order.

Hello,

I see that for every body, it's working but it does not work for me.
I'm on Magento CE 2.3.3. I just followed this documentation https://github.com/magento/baler/blob/master/docs/ALPHA.md .
My result is that bundles are well created. And the core-bundle js is called. But at the same time, I have many other js files which are called.
I don't know what's wrong ... Maybe it's because of custom codes, on what do we need to be careful ?

Then I try to run baler in my root Magento project it stuck with such error:
Error: ENOTDIR: not a directory, scandir ' .. /app/code/.DS_Store'

Yes, I use Mac. To reproduce on Linux try to create the same dir with "."

trace output:

(450.8348079994321): Resolving dependency "jquery/jquery.mobile.custom" from "unknown source"
(450.95511799678206): Resolving dependency "mage/common" from "unknown source"
(451.0416179969907): Resolving dependency "mage/dataPost" from "unknown source"
[...]
(452.7179719991982): Resolving dependency "Magento_Catalog/js/more-less" from "unknown source"
(452.83620499819517): Preparing to analyze "jquery/jquery.mobile.custom" for dependencies
(453.26991099864244): Warning for missing dependency "jquery/jquery.mobile.custom", which was required by "<entry point>"
(453.3077109977603): Preparing to analyze "mage/common" for dependencies
(453.3644319996238): Warning for missing dependency "mage/common", which was required by "<entry point>"
(453.37505899742246): Preparing to analyze "mage/dataPost" for dependencies
(453.4178909994662): Warning for missing dependency "mage/dataPost", which was required by "<entry point>"
[...]
(468.5774130001664): Resolving dependency "Magento_Catalog/js/more-less" from "unknown source"

baler then gets stuck in an endless loop.

For the Graph Preloading functionality, it's necessary for baler to spit out a mapping of JavaScript dependencies needed per .phtml file.

Requirements

1. Mapping file should be named baler-preload-map.json, and should be written to the root of a locale dir, as a sibling of requirejs-baler-config.js
2. A JSON Schema should be provided for Magento_Baler to use for validation
3. Key/Value should be
   • Key: Path from store root to a .phtml file
   • Value: Array of relative paths to JavaScript files from the require basedir

Example

{
   "vendor/magento/module-example/view/frontend/templates/some-template.phtml": ["path/to/file/1.js", "path/to/file/2.js"],
   "vendor/thirdparty/module-bar/view/frontend/templates/another-template.phtml": ["another/file.js", "yet/another/file/2.js"]
}

Question: Can this speed up Magento Admin too?

 baler build --trace
✔ Collected theme/module data 142ms
⠋ (Magento/luma) Create core bundle file⠋ (Etilize/import) Create core bundle file
Optimization is done, but stats have not been implemented in the CLI yet
⠋ (Etilize/import) Create core bundle file

....
(2492.643615990877): Preparing to analyze "MutationObserver" for dependencies
(2496.74079400301): Preparing to analyze "FormData" for dependencies
(2507.8036310076714): Resolving dependency "jquery/jquery.mobile.custom" from ""
(2508.143660992384): Resolving dependency "jquery" from ""
(2508.2522000074387): Resolving dependency "(2482.613610982895): Resolving dependency "jquery" from "jquery-ui-modules/datepicker"
(2482.7034960091114): Resolving dependency "jquery-ui-modules/core" from "jquery-ui-modules/datepicker"
(2482.7716360092163): Preparing to analyze "jquery-ui-modules/slider" for dependencies
(2491.183733999729): Discovered dependencies for "jquery-ui-modules/slider": jquery, jquery-ui-modules/core, jquery-ui-modules/mouse
(2491.22227999568): Resolving dependency "jquery" from "jquery-ui-modules/slider"
(2491.291195988655): Resolving dependency "jquery-ui-modules/core" from "jquery-ui-modules/slider"
(2491.3486459851265): Resolving dependency "jquery-ui-modules/mouse" from "jquery-ui-modules/slider"
(2492.643615990877): Preparing to analyze "MutationObserver" for dependencies
(2496.74079400301): Preparing to analyze "FormData" for dependencies
(2507.8036310076714): Resolving dependency "jquery/jquery.mobile.custom" from ""
(2508.143660992384): Resolving dependency "jquery" from ""
(2508.2522000074387): Resolving dependency "ns" from ""
(2508.3212589919567): Resolving dependency "vmouse" from ""
(2508.3784649968147): Resolving dependency "support/touch" from ""
(2508.493703007698): Resolving dependency "mage/common" from ""
(2508.5813890099525): Resolving dependency "domReady" from ""
(2508.661637991667): Resolving dependency "mage/dataPost" from ""
(2508.7471689879894): Resolving dependency "mage/template" from ""
(2508.830219000578): Resolving dependency "underscore" from ""
(2508.9048919975758): Resolving dependency "Magento_Ui/js/modal/confirm" from ""
(2509.0140880048275): Resolving dependency "mage/translate" from ""
(2509.0956659913063): Resolving dependency "mage/mage" from ""
(2509.1743820011616): Resolving dependency "mage/apply/main" from ""
(2509.274531006813): Resolving dependency "mage/apply/scripts" from ""
(2509.363907009363): Resolving dependency "jquery/jquery-ui" from ""
(2509.44534099102): Resolving dependency "Magento_Ui/js/modal/modal" from ""
(2509.5548349916935): Resolving dependency "text!ui/template/modal/modal-popup.html" from ""
(2509.5758909881115): Resolving dependency "text" from ""
(2509.7167730033398): Resolving dependency "text" from ""
(2509.785418987274): Resolving dependency "text!ui/template/modal/modal-slide.html" from ""
(2509.8036189973354): Resolving dependency "text" from ""
(2510.175595998764): Resolving dependency "text!ui/template/modal/modal-custom.html" from ""
(2510.2254819869995): Resolving dependency "text" from ""
(2510.3711499869823): Resolving dependency "Magento_Ui/js/lib/key-codes" from ""
(2510.4803000092506): Resolving dependency "mage/bootstrap" from ""
(2510.565497994423): Resolving dependency "Magento_Ui/js/lib/knockout/bootstrap" from ""
(2510.691917002201): Resolving dependency "knockoutjs/knockout" from ""
(2510.7790269851685): Resolving dependency "Magento_Ui/js/lib/knockout/template/engine" from ""
(2510.903595983982): Resolving dependency "Magento_Ui/js/lib/knockout/template/observable_source" from ""

normally it says unknown source but I changed it to say ""..

nevermind it was newer version of jquery.mobile.custom.js causing failure

I tried to use
npm install
in the root folder of baler.
And I've got an error:
npm ERR! Maximum call stack size exceeded

I'm aware that this project is still in its infancy but I'm pleased to see it developing so well!

When running baler in the root of my Magento 2 store, I get an immediate error

Error: ENOENT: no such file or directory, open

complaining that '/etc/module.xml' does not exist in one of our third-party modules in the vendor directory. This is true because this vendor has decided to place their 2 Magento 2 modules in the same composer package. Therefore the path to each should be 'vendor/vendorname/modulename/src/module1/etc/module.xml' etc.

If this is a complicated issue to solve then some documented way to disable some modules from being bundled would also be great.

Request to loosen up findMagentoRoot check requirements.

Currently Baler checks for these required directories:

• app/
• pub/
• lib/
• index.php

However, index.php is not necessary, while vendor probably is necessary (for the graph building later on).

The baler build should preferably only check for pub/static/frontend because that's the only place it needs to get data from, and it's enough to determine you are in the right directory.

For the baler graph method (when released) you could do a wider check, including the other directories.

1 - The error message could be more descriptive (currently says Could not find Magento store root from [X].
2 - The requirements could be reduced

Why is this important?

A lot of us devs out there use docker and run npm commands locally, not in the containers. So we usually don't have all these directories locally mounted/synced to the containers.
Especially lib/ and index.php are very rare to have synced.

It seems that baler needs a PHP executable. This is not documented. Of course if you work with Magento you usually also have PHP installed, but I was building a Docker image for baler and didn't realize that PHP was also directly required for executing baler.

It would be great, if the documentation included that information.

Also why does baler need a PHP executable...?

When a theme is bundled, the following artifacts are written to disk:

• requirejs-baler-config.js
• balerbundles/core-bundle.js
• baler-preload-map.json

The bundler works by collecting a list of all locales for a single theme, but only performs analysis and bundling of the first one found on disk.

Right now, other locales do not get bundling files.

Acceptance Criteria

This story is complete when the artifacts referenced above are copied to each locale.

Example: Using luma, with en_US, es_ES, and fr_BE. In this example, bundling will run against luma/en_US, and then artifacts will be copied to luma/es_ES and fr_BE.

I have installed properly baler module on Magento 2.3.4 and its also working but this is not including define or require in bundles and giving errors

mixins.js:5 Uncaught ReferenceError: define is not defined
at mixins.js:5
(anonymous) @ mixins.js:5
jquery.mobile.custom.js:43 Uncaught TypeError: Cannot read property 'event' of undefined
at jquery.mobile.custom.js:43
at jquery.mobile.custom.js:37
at jquery.mobile.custom.js:20
at jquery.mobile.custom.js:11
(anonymous) @ jquery.mobile.custom.js:43
(anonymous) @ jquery.mobile.custom.js:37
(anonymous) @ jquery.mobile.custom.js:20
(anonymous) @ jquery.mobile.custom.js:11
(index):740 Uncaught TypeError: require.config is not a function
at (index):740
(anonymous) @ (index):740
(index):775 Uncaught TypeError: require is not a function
at (index):775
(anonymous) @ (index):775
(index):793 Uncaught TypeError: require is not a function
at

(index):793

Console message after "npm run build". Logs are attached. Dependencies were updated, tests were passed well.

@magento/[email protected] build /Users/myName/Sites/baler
tsc

src/createBundleFromDeps.ts:15:28 - error TS2307: Cannot find module './debug'.

15 import { debugEvent } from './debug';
~~~~~~~~~
Found 1 error.

npm ERR! code ELIFECYCLE
npm ERR! errno 2
npm ERR! @magento/[email protected] build: tsc
npm ERR! Exit status 2
npm ERR!
npm ERR! Failed at the @magento/[email protected] build script.
npm ERR! This is probably not a problem with npm. There is likely additional logging output above.

npm ERR! A complete log of this run can be found in:
npm ERR! /Users/myName/.npm/_logs/2019-09-06T07_07_19_540Z-debug.log
2019-09-06T07_07_19_540Z-debug.log

On our production environment we don't have the possibility to use node, so I wonder if I can run baler on our development machine (which does run node), and use the results on our production environment..?

Hi,

I've been trying to implement Baler on one of our Magento instances. Nearly got it working, only the storage API is not being loaded correctly. This is breaking multiple JS files. Is this a known issue?

I tried to run npm test on a Macbook (node v10.16.0, npm 6.9.0), but failed. Part of the log is as follows.

12 info lifecycle @magento/[email protected]~format:check: Failed to exec format:check script
13 verbose stack Error: @magento/[email protected] format:check: `prettier --list-different $(head -1 prettier-globs)`
13 verbose stack Exit status 1
13 verbose stack     at EventEmitter.<anonymous> (/usr/local/lib/node_modules/npm/node_modules/npm-lifecycle/index.js:301:16)
13 verbose stack     at EventEmitter.emit (events.js:198:13)
13 verbose stack     at ChildProcess.<anonymous> (/usr/local/lib/node_modules/npm/node_modules/npm-lifecycle/lib/spawn.js:55:14)
13 verbose stack     at ChildProcess.emit (events.js:198:13)
13 verbose stack     at maybeClose (internal/child_process.js:982:16)
13 verbose stack     at Process.ChildProcess._handle.onexit (internal/child_process.js:259:5)
14 verbose pkgid @magento/[email protected]

Unsure whether this is or isn't a Baler issue, or something modulemakers could be doing wrong.

When running 'baler/bin/baler' from the root i get the following message:
Error: ENOENT: no such file or directory, open [domain]/vendor/mirasvit/module-helpdesk/etc/module.xml

My assumption here is that baler is spitting through modules to find dependencies and possible JS files to bundle?

This is actually correct though, the location it should be looking for is vendor/mirasvit/module-helpdesk/src/Helpdesk/etc/module.xml. I believe the module maps these locations through composer.json:

It looks like there's an assumption about where baler should find vendor files, based on the generated files? This assumption then doesn't take into account that modulemakers might map their modules in a different way.

I could be totally off-base here, but hope this helps make baler the tool we'll all be using in a month (or two ;))

Hey there,
Baler hit the following error on the mirasvit elastic search ultimate module:

Error: Unable to locate "module.xml" for the module at path "vendor/mirasvit/module-search-elastic-ultimate"
    at findRealModuleRoot (/Users/scottfriend/Sites/baler/src/magentoFS.ts:241:15)
    at processTicksAndRejections (internal/process/task_queues.js:89:5)
    at getModuleConfig (/Users/scottfriend/Sites/baler/src/magentoFS.ts:204:28)
    at async Promise.all (index 154)
    at getComposerComponents (/Users/scottfriend/Sites/baler/src/magentoFS.ts:332:18)
    at async Promise.all (index 0)
    at Object.getComponents (/Users/scottfriend/Sites/baler/src/magentoFS.ts:62:37)
    at async Promise.all (index 1)
    at Object.collectStoreData (/Users/scottfriend/Sites/baler/src/collectStoreData.ts:15:58)
    at Object.run (/Users/scottfriend/Sites/baler/src/cli.ts:20:19)

This composer module doesn't actually contain a Magento Module. It's just a shell of a module to pull in other separate actual Magento modules via the compser.json it contains. Kinda goofy, but I think it allows Mirasvit to sell a module that is really just a bundle of other modules.

The contents of the directory are:

• vendor/mirasvit/module-search-elastic-ultimate
  • CHANGELOG.md
  • composer.json

Maybe baler could just skip over modules that don't contain a valid module.xml file?