stefanpenner / broccoli-concat-analyser Goto Github PK

License: MIT License

JavaScript 12.17% HTML 87.83%

broccoli-concat-analyser's Introduction

broccoli-concat-analyser

When using a build chain based on broccoli-concat, this tool will show you the size and contents of the concatenated bundles, including:

raw size
uglified size (for JS right now)
compressed size (including uglify if applicable)

Interactive Demo

Ember addon

For usage in an Ember app, ember-cli-bundle-analyzer wraps this library as an Ember addon and integrates it more tightly into your development workflow.

See the instructions on its documentation page.

Usage

To directly use this tool, follow these steps:

npm install -g broccoli-concat-analyser
CONCAT_STATS=true ember s
broccoli-concat-analyser ./concat-stats-for
open ./concat-stats-for/index.html in any browser for the foamtree interactive map

Notes:

Be sure to have the latest version of broccoli-concat installed
The process output can be found in the .out.json files in the directory: ./concat-stats-for
Remove the ./concat-stats-for directory and run step 2 after any changes for an updated output

Features

got ideas? Submit PRs!

broccoli-concat-analyser's People

Contributors

Stargazers

Watchers

Forkers

calderas simonihmig kimek betocantu93 tomoguisuru chadhietala scalvert carterchung rajasegar robertomalatesta lifeart

broccoli-concat-analyser's Issues

Add full module path to tooltip

Having the module path easily available alongside the existing size values could be helpful, e.g. to exclude that module by adding it to ember-cli-funnel's exclude list: https://github.com/kellyselden/ember-cli-funnel#usage

Fails on some files

Throws errors like these on OS X with Node 7.10.0:

undefined:1625
    throw new JS_Parse_Error(message, filename, line, col, pos);
    ^
SyntaxError: Unexpected token operator «*», expected punc «(»
    at JS_Parse_Error.get (eval at <anonymous> (/usr/local/lib/node_modules/broccoli-concat-analyser/node_modules/uglify-js/tools/node.js:1:1), <anonymous>:86:23)

Inaccurate total zip size

I noticed the total zipped size of an asset (say vendor.js) as shown in the analyser output is much bigger that the real one.

Which is understandable given how we compute that number: we just compute the sum of all (individually zipped) sizes of the modules it contains. However zip/deflate (especially its duplicate string elimination) will work much better on a large concatenated file containing all modules, than when applied to all single modules individually. Which explains the difference AFAICT.

Maybe we should compute the zipped size of the whole asset file, and "spread" that to its individual modules proportionally (like zipped module_size = total_zipped_size / total_minified_size * minified_module_size)?

I think its inherent to the compression algorithm that we cannot accurately determine how much a given module will attribute to the total (concatenated) zipped size, but this could be a good enough approximation, that makes sure that the sum of all zipped module sizes equals the real zipped size of the concatenated file.

@stefanpenner the changes required for this are probably not very difficult, I can try to bring up a PR when we agree on this!

Improved README.md

Could it possible to improve the current readme? It's not quite clear to me what the steps are to actually get the process output.

0 npm install -g broccoli-concat-analyser
1 using broccoli-concat (latest versions)
2 CONCAT_STATS=true ember s
3 broccoli-concat-analyser ./concat-stats-for
4 look at .out.json files in ./concat-stats-for for the process output

1 It's a dependency of ember-cli, so I can ignore that probably unless someone doesn't have it installed.
2 Where do you put CONCAT_STATS=true. This does not work, I just found out this is a linux style command. I'm on Windows.
3 What does this mean?
4 I can't find the file, probably because I'm not doing it correctly.

Aggregated uglified size of non-uglifyable assets is wrong

See https://github.com/stefanpenner/broccoli-concat-analyser/blob/master/test/fixtures/output/build-output-summary.json#L169

PR following...

ember-fetch appears multiple times

ember-fetch is deduped by NPM:

I will add some more notes later, ember-fetch seems to be present only once in the resulting bundle and sourcemap.

Uglify errors should throw

uglify-es does not throw an error itself, so in case of minification errors the root issue is swallowed and results in a TypeError: Cannot read property 'length' of undefined in here https://github.com/stefanpenner/broccoli-concat-analyser/blob/master/lib/summarize.js#L54.

Cannot read property 'length' of undefined

Livereload server on https://localhost:49155
The Broccoli Plugin: [BroccoliMergeTrees: styles] failed with:
TypeError: Cannot read property 'length' of undefined
    at /Users/bas/Documents/Projects/PLHW.ProjectHealthyFeet/applications/hf-app-lab/node_modules/broccoli-concat/lib/strategies/simple.js:128:40
    at Array.reduce (native)
    at SimpleConcat.fileSizes (/Users/bas/Documents/Projects/PLHW.ProjectHealthyFeet/applications/hf-app-lab/node_modules/broccoli-concat/lib/strategies/simple.js:127:7)
    at Concat._doPatchBasedBuild (/Users/bas/Documents/Projects/PLHW.ProjectHealthyFeet/applications/hf-app-lab/node_modules/broccoli-concat/concat.js:152:33)
    at Concat.build (/Users/bas/Documents/Projects/PLHW.ProjectHealthyFeet/applications/hf-app-lab/node_modules/broccoli-concat/concat.js:104:17)
    at /Users/bas/Documents/Projects/PLHW.ProjectHealthyFeet/applications/hf-app-lab/node_modules/broccoli-plugin/read_compat.js:93:34
    at tryCatch (/Users/bas/Documents/Projects/PLHW.ProjectHealthyFeet/applications/hf-app-lab/node_modules/rsvp/dist/rsvp.js:525:12)
    at invokeCallback (/Users/bas/Documents/Projects/PLHW.ProjectHealthyFeet/applications/hf-app-lab/node_modules/rsvp/dist/rsvp.js:538:13)
    at publish (/Users/bas/Documents/Projects/PLHW.ProjectHealthyFeet/applications/hf-app-lab/node_modules/rsvp/dist/rsvp.js:508:7)
    at flush (/Users/bas/Documents/Projects/PLHW.ProjectHealthyFeet/applications/hf-app-lab/node_modules/rsvp/dist/rsvp.js:2415:5)

"CONCAT_STATS=true ember s" is not generating /concat-stats-for dir

CONCAT_STATS=true ember s is not generating /concat-stats-for dir

v4.3.5

Breaks with ember-cli-chart

Hello guys,

I'm trying to run this on a project of mine but it breaks if I have the ember-cli-chart addon installed.

To reproduce:

ember new testApp
ember install ember-cli-chart
CONCAT_STATS=true ember s
broccoli-concat-analyser ./concat-stats-for

The error is:

⠹ processing: ./concat-stats-for2-vendor.js.json > ./concat-stats-for2-vendor.js.out.jsonfs.js:641
  return binding.open(pathModule._makeLong(path), stringToFlags(flags), mode);
                 ^

Error: ENOENT: no such file or directory, open 'concat-stats-for/2-vendor.js/home/t318791/projects/testApp/node_modules/chart.js/dist/Chart.js'
    at Error (native)
    at Object.fs.openSync (fs.js:641:18)
    at Object.fs.readFileSync (fs.js:509:33)
    at summary.files.Object.keys.map.relativePath (/home/t318791/.nvm/versions/node/v6.11.0/lib/node_modules/broccoli-concat-analyser/index.js:28:22)
    at Array.map (native)
    at summarize (/home/t318791/.nvm/versions/node/v6.11.0/lib/node_modules/broccoli-concat-analyser/index.js:27:38)
    at entries.forEach.entry (/home/t318791/.nvm/versions/node/v6.11.0/lib/node_modules/broccoli-concat-analyser/index.js:57:3)
    at Array.forEach (native)
    at Object.<anonymous> (/home/t318791/.nvm/versions/node/v6.11.0/lib/node_modules/broccoli-concat-analyser/index.js:54:9)
    at Module._compile (module.js:570:32)

I'm using

$ ember -v
ember-cli: 2.13.3
node: 6.11.0
os: linux x64

Output to SVG?

Let's say I want to include the very nice bundle analysis in my repo and/or as artifacts on my ci/cd environment..

Is there a way I can output a self-contained file?
at the moment, there aren't many files,

but without zipping, it'd be hard to send this to somebody.

I can screenshot,

but I lose the ability to zoom in on the smaller pieces -- so I think an SVG would be the perfect solution to this. :)

group by module paths (to help identify offending addons/areas)

so given

ember-data/-private/system/store
ember-data/adapters/rest.js

We could create the following depth:

ember-data
- adapters
  - rest.js
- -private
  - system
    - store.js

Most likely though, merely adding 1 additional grouping (the top level) would drastically improve this.

ember-data
- adapters/rest.js
- -private/system/store.js

does not work with canary (49e58e45604bd63ac1001cb491195c032c050053 (August 16, 2018))

I've been using the analyzer for fun over at https://emberclear.io/bundle.html
:)

But the generated bundle is a smidge out of date, as recently (I believe) there has been some private api change that the analyzer was relying on?

Here is my package.json for reference: https://gitlab.com/NullVoxPopuli/emberclear/blob/master/packages/frontend/package.json

windows CI

Excludes packages imported using ember-auto-import

It appears that broccoli-concat-analyser doesn't include packages which have been imported using ember-auto-import. I'm unsure whether the issue is in this addon or ember-auto-import but thought I'd open it here first.

To reproduce the issue I've made a sample repo where I've done the following:

Installed ember-auto-import
Installed lodash
Imported lodash into app/app.js

Running ember asset-sizes we get the following size vendor.js

File sizes:
 - dist/assets/vendor.js: 3.2 MB (683.25 KB gzipped)

But broccoli-concat-analyser outputs the following size for vendor.js

 - dist/assets/vendor.js: 2.65 MB (218.44 KB gzipped)

There looks to be a large discrepancy there. Looking at the generated stats it appears the auto-imported lodash has been excluded.

The reproduction repo is here and I've committed the generated broccoli-concat-analyser stats

TypeError: "buffer" argument must be a string, Buffer, TypedArray, or DataView

I'm not able to run this anymore. I get this:

processing: ./concat-stats-for11-app.js.json > ./concat-stats-for11-app.js.out.jsonzlib.js:138
    throw new TypeError('"buffer" argument must be a string, Buffer, ' +
    ^

TypeError: "buffer" argument must be a string, Buffer, TypedArray, or DataView
    at zlibBufferSync (zlib.js:138:11)
    at Object.deflateSync (zlib.js:588:14)
    at summary.files.Object.keys.map.relativePath (/Users/rwwagner/.nvm/versions/node/v8.7.0/lib/node_modules/broccoli-concat-analyser/lib/summarize.js:22:25)
    at Array.map (<anonymous>)
    at summarize (/Users/rwwagner/.nvm/versions/node/v8.7.0/lib/node_modules/broccoli-concat-analyser/lib/summarize.js:16:38)
    at entries.forEach.entry (/Users/rwwagner/.nvm/versions/node/v8.7.0/lib/node_modules/broccoli-concat-analyser/lib/summarize-all.js:13:5)
    at Array.forEach (<anonymous>)
    at summarizeAll (/Users/rwwagner/.nvm/versions/node/v8.7.0/lib/node_modules/broccoli-concat-analyser/lib/summarize-all.js:9:11)
    at Object.<anonymous> (/Users/rwwagner/.nvm/versions/node/v8.7.0/lib/node_modules/broccoli-concat-analyser/lib/cli.js:19:1)
    at Module._compile (module.js:624:30)

cleanup lolcode introduced by: #1

addon.js vendor/addon.js seem funky?

Application css is missing

When viewing the visualization, I'm noticing that the application css is completely missing - there's no area for it at all. The app js, vendor js, and vendor css are all showing up fine.

I've tried this on both a large application with lots of css, and a small add-on with just a bit.

Tested with ember-cli 2.18 and 3.0, as well as broccoli-concat-analyser 2.1.0 and 3.0.0.

Let me know what further info you need. Thanks!

issues with lazy engines

[node 11/03/16 10:29:12]   return binding.open(pathModule._makeLong(path), stringToFlags(flags), mode);
[node 11/03/16 10:29:12]                  ^
[node 11/03/16 10:29:12] 
[node 11/03/16 10:29:12] Error: ENOENT: no such file or directory, open 'concat-stats-for/undefined-engine-vendor.js/engines-dist/messaging-ext/assets/engine-vendor.js'
[node 11/03/16 10:29:12]     at Error (native)
[node 11/03/16 10:29:12]     at Object.fs.openSync (fs.js:549:18)
[node 11/03/16 10:29:12]     at Object.fs.readFileSync (fs.js:397:15)
[node 11/03/16 10:29:12]     at /Users/syeung/Documents/Development/new/voyager-web_trunk/node_modules/broccoli-concat-analyser/index.js:27:22
[node 11/03/16 10:29:12]     at Array.map (native)
[node 11/03/16 10:29:12]     at summarize (/Users/syeung/Documents/Development/new/voyager-web_trunk/node_modules/broccoli-concat-analyser/index.js:26:38)
[node 11/03/16 10:29:12]     at /Users/syeung/Documents/Development/new/voyager-web_trunk/node_modules/broccoli-concat-analyser/index.js:56:3
[node 11/03/16 10:29:12]     at Array.forEach (native)
[node 11/03/16 10:29:12]     at Object.<anonymous> (/Users/syeung/Documents/Development/new/voyager-web_trunk/node_modules/broccoli-concat-analyser/index.js:53:9)
[node 11/03/16 10:29:12]     at Module._compile (module.js:410:26)

Ember.js appears multiple times

https://emberclear.io/bundle/broccoli.html

Use terser instead of uglify-es

uglify-es is unmaintained. Also broccoli-uglify-sourcemap has switched to terser as well.

This hopefully might fix some errors here as well, when uglify-es failed to minify some code (reproducible with https://github.com/NYCPlanning/labs-zola)

add parallelism support

When uglifying, we end up being quite CPU bound, and large projects likely suffer.

Using:

worker pool -> https://github.com/josdejong/workerpool
and maybe async-promise-queue -> https://github.com/stefanpenner/async-promise-queue

We can likely improve throughput. That being said, it will require some testing and tuning for sure.

Broken when used with Ember 2.14

I believe this addon relies on vendor/ember/ember.debug.map being present.

There is an outstanding issue here emberjs/ember.js#15464 that I believe is related. I'm not sure of a workaround atm, but thought this might be a good place to add insight into implications of sourcemaps not being available.