wearejh / config-gen Goto Github PK

View Code? Open in Web Editor NEW

84.0 84.0 9.0 419 KB

Generate RequireJS Optimizer configuration for a Magento 2 website based on real-world usage.

Rust 99.74% Shell 0.11% JavaScript 0.15%

config-gen's People

Contributors

Stargazers

Watchers

Forkers

vkerkhoff tdgroot januszpl beejhuff mrlovelies muhammedv davidalger thaopw shakyshane

config-gen's Issues

build.json should be optimize=uglify by default

the default setup should give the best perf possible

More verbose logging

I'd like to see exactly what the app is doing. What it is adding and why. What page it is proxying from my browser.

I want all of that in stdout so I can see it as I'm working with it.

actix_web::pipeline: Error occurred during request handling: err

Upon first navigating to a slug belonging to a bundle registered in bundle-config.yml in order to build configuration for it, I get the following messages in the terminal:

ERROR 2019-08-29T09:47:56Z: actix_web::pipeline: Error occurred during request handling: err
ERROR 2019-08-29T09:47:56Z: actix_web::server::h2: Unhandled error: err

After which no changes are reflected at the endpoints for build.js, seed.js, etc.

This is what my config looks like:

bundles:
  - name: "bundles/main"
    urls:
      - "/"
    children:
      - name: "bundles/product"
        urls:
          - "/index.php/test-product"
        children: []
      - name: "bundles/category"
        urls:
          - "/index.php/test-category"
        children: []
      - name: "bundles/checkout"
        urls:
          - "/index.php/checkout"
        children: []

should support high-level integration tests

Ideally something along these lines (pseudo code)

#[test]
fn test_api() {
   run_from_args(vec!["http://example.com"], "/build.json",  |(sys, address, response)| {
       // assert on response here
   })
}

Config json / yml tree - build.json doesn't have all bundles ?

Not sure if I create config tree properly but when experimenting I saw it stop generated other bundles in build.json. I've also tested sample config and added children to bundles/product

{
  "name": "bundles/basket2",
  "urls": [
    "/index.php/checkout/cart2/",
    "/index.php/checkout/cart2"
  ],
  "children": [
    {
      "name": "bundles/checkout2",
      "urls": [
        "/index.php/checkout2/"
      ],
      "children": [

      ]
    }
  ]
}

so now sample config looks like:

{
  "bundles": [
    {
      "name": "bundles/main",
      "urls": [
        "/",
        "/index.php/women/tops-women/jackets-women.html"
      ],
      "children": [
        {
          "name": "bundles/product",
          "urls": [
            "/index.php/juno-jacket.html"
          ],
          "children": [
          {
              "name": "bundles/basket2",
              "urls": [
                "/index.php/checkout/cart2/",
                "/index.php/checkout/cart2"
              ],
              "children": [
                {
                  "name": "bundles/checkout2",
                  "urls": [
                    "/index.php/checkout2/"
                  ],
                  "children": [

                  ]
                }
              ]
            }
          ]
        },
        {
          "name": "bundles/basket",
          "urls": [
            "/index.php/checkout/cart/",
            "/index.php/checkout/cart"
          ],
          "children": [
            {
              "name": "bundles/checkout",
              "urls": [
                "/index.php/checkout/"
              ],
              "children": [

              ]
            }
          ]
        }
      ]
    }
  ]
}

and the build.json:

{
  "generateSourceMaps": true,
  "inline_text": true,
  "optimize": "none",
  "deps": [],
  "map": {},
  "config": {},
  "shim": {},
  "paths": {},
  "modules": [
    {
      "name": "requirejs/require",
      "include": [],
      "exclude": [],
      "create": false
    },
    {
      "name": "bundles/main",
      "include": [],
      "exclude": [
        "requirejs/require"
      ],
      "create": true
    },
    {
      "name": "bundles/product",
      "include": [],
      "exclude": [
        "requirejs/require",
        "bundles/main"
      ],
      "create": true
    },
    {
      "name": "bundles/basket2",
      "include": [],
      "exclude": [
        "requirejs/require",
        "bundles/main",
        "bundles/product"
      ],
      "create": true
    },
    {
      "name": "bundles/checkout2",
      "include": [],
      "exclude": [
        "requirejs/require",
        "bundles/main",
        "bundles/product",
        "bundles/basket2"
      ],
      "create": true
    }
  ]
}

it missing bundles/basket and bundles/checkout

requirejs-config post errors if it follows source-map declaration

Should not include to `baseUrl` or `dir` props in the build output build.json

Should add dep "jquery/jquery.cookie" to `jquery/jquery-migrate in shim

should forward http status codes

should support self-signed certs on the proxy target (for local development)

API errors should have correct status code

should replace external paths with `empty:`

need to document options

Things that are supported, but not yet documented:

basic auth for proxy target
module black_list
--port
disable browser cache by opening chrome with --disk-cache-size=1

If the proxy sends a HTTP request to https://example.com, it's possible that https://example.com might try to redirect, to something like https://example.com/uk/ - but I've noticed on some Magento stores, this location header might be for an insecure url (which will subsequently be upgraded), eg:

GET https://example.com

resp:

302
...
"location": "http://example.com/uk/"
...

The problem is that the proxy will rewrite that location address to be something like http://127.0.0.1:9000/uk/ - where the scheme is downgraded and since we're not listening over http, the connection will appear to hang.

solution:

when rewriting "location": "http://example.com/uk/" - ensure we always match the scheme of the proxy regardless

crates.io

Is this program available on crates.io? I couldn't find it there.

should parse entire magento-generated requirejs-config file

It turned out, you cannot ask requireJS in the browser for the correct 'merged' config as it does some collapsing at run time that renders it invalid for the build process (eg: dropping some top-level 'deps')

Instead, just parse the entire file pulling out only the config

Should always bind to a random, available port unless overriden with `--port`

Address already in use

I'm using the Ubuntu build and getting this;

export RUST_BACKTRACE=1 && ./config-gen http://example.com/ --config config-gen.yml
thread 'main' panicked at 'Couldn't start the application: Os { code: 98, kind: AddrInUse, message: "Address already in use" }', libcore/result.rs:1009:5
stack backtrace:
   0: std::sys::unix::backtrace::tracing::imp::unwind_backtrace
             at libstd/sys/unix/backtrace/tracing/gcc_s.rs:49
   1: std::sys_common::backtrace::print
             at libstd/sys_common/backtrace.rs:71
             at libstd/sys_common/backtrace.rs:59
   2: std::panicking::default_hook::{{closure}}
             at libstd/panicking.rs:211
   3: std::panicking::default_hook
             at libstd/panicking.rs:227
   4: std::panicking::rust_panic_with_hook
             at libstd/panicking.rs:476
   5: std::panicking::continue_panic_fmt
             at libstd/panicking.rs:390
   6: rust_begin_unwind
             at libstd/panicking.rs:325
   7: core::panicking::panic_fmt
             at libcore/panicking.rs:77
   8: core::result::unwrap_failed
   9: config_gen::main
  10: std::rt::lang_start::{{closure}}
  11: std::panicking::try::do_call
             at libstd/rt.rs:59
             at libstd/panicking.rs:310
  12: __rust_maybe_catch_panic
             at libpanic_unwind/lib.rs:102
  13: std::rt::lang_start_internal
             at libstd/panicking.rs:289
             at libstd/panic.rs:392
             at libstd/rt.rs:58
  14: main
  15: __libc_start_main
  16: _start

ci: should publish on pushes to master

Cannot find key.pem file

When I try and run the binary downloaded from the release tab I get an error saying cannot find src/key.pm

This is the full error:
thread 'main' panicked at 'called Result::unwrap()on anErrvalue: ErrorStack([Error { code: 33558530, library: "system library", function: "fopen", reason: "No such file or directory", file: "bss_file.c", line: 406, data: "fopen(\'src/key.pem\',\'r\')" }, Error { code: 537346050, library: "BIO routines", function: "FILE_CTRL", reason: "system lib", file: "bss_file.c", line: 408 }, Error { code: 336265218, library: "SSL routines", function: "SSL_CTX_use_PrivateKey_file", reason: "system lib", file: "ssl_rsa.c", line: 633 }])', libcore/result.rs:945:5

shoudl support blacklist of modules to NEVER include in build or loaders

should match case-insensitve on header values

`inline_text` should be `inlineText` in build.json

Should create blank requirejs module to include 'deps'

`loaders.js` should give either an error or blank file when no additional bundles are configured

Error when running r_js XMLHttpRequest is not defined

I receive the below error when running:

sudo r_js -o build.json \
    baseUrl=pub/static/frontend/MySite/default/en_GB_src \
    dir=pub/static/frontend/MySite/default/en_GB \
    optimize=none

Tracing dependencies for: requirejs/require
ReferenceError: XMLHttpRequest is not defined
In module tree:
    mage/dataPost
      Magento_Ui/js/modal/confirm
        Magento_Ui/js/modal/modal
          text

Error: ReferenceError: XMLHttpRequest is not defined
In module tree:
    mage/dataPost
      Magento_Ui/js/modal/confirm
        Magento_Ui/js/modal/modal
          text

This appears to be caused by XMLHttpRequest being a Web API and not a Node API.

ci: build is currently failing due to openSSL version on travis

should not include 'requirejs/require' in 'loaders' output

Running ./config-gen on osx

Not quite related to config-gen. more to os x and permission/zsh/rustc issue?

I have permission denied when running
./config-gen

Anyone had similar issues?

preset: m2 - should allow pre-populating existing config

text module needs replacing

bash: ./config-gen: is a directory

when i run './config-gen http://example.com --config config-gen.yml' or './config-gen', console warning: bash: ./config-gen: is a directory

Should support a blacklist of modules that are too 'broken' to be included

Add ubuntu release

Did a release for ubuntu:

https://github.com/robin31/config-gen/releases/

Should not include js-translation in bundle

should not enforce `yaml` for config files, should also allow `json`

Doesn't release port when application is shutdown

separate parsing of requirejs-config.js to enable use in `wasm`

Creating files with depedencies list per page

It would be great if we could add here creation of formatted and filtered dependencies list per page in format like (with removed mixins)
https://devdocs.magento.com/guides/v2.3/performance-best-practices/advanced-js-bundling.html#6-format-and-filter-the-output - plus per page loader.js file (with extra file loaders-pageType.js with list of those depedencies for further manual edits)

and if possible also create
Depdencies distrubition file:
https://devdocs.magento.com/guides/v2.3/performance-best-practices/advanced-js-bundling.html#8-create-a-dependencies-distribution-file

to help with creating manual bundles.

Of course I can use phantomjs, manualy create it, or using other way/tool/script - but it would be nice if we had this extra files created in this tools for checking alternative - manual bundles

actix_web pipeline error - 502 - Too many open files

When experimenting with different bundle-config.yml I was editing config and browsing website. Then suddenly I got empty build.json and I noticed that I had serveral errors like this one:

ERROR 2018-11-21T01:21:50Z: actix_web::pipeline: Error occured during request handling, status: 502 Bad Gateway Failed to connect to host: Too many open files (os error 24)

and then:

thread 'arbiter:2fed453f-22f1-4ccf-b941-935d1a843a2b:actix-net-worker-0' panicked at 'can read current dir: Os { code: 24, kind: Other, message: "Too many open files" }', libcore/result.rs:1009:5
note: Run with `RUST_BACKTRACE=1` for a backtrace.
ERROR 2018-11-21T01:21:50Z: actix_net::server::accept: No workers
ERROR 2018-11-21T01:21:50Z: actix_net::server::server: Worker has died 0, restarting

Adding this ticket for further investigating.

My config when error happened (custom project)

bundles:
  - name: "bundles/simple-page"
    urls:
      - "/o-nas"
    children:
      - name: "bundles/cms-page"
        urls:
        - "/kontakt"
        children:
          - name: "bundles/category"
            urls:
              - "/meskie/odziez/spodnie"
              - "/search/result/?q=spodnie"
            children: []
          - name: "bundles/homepage"
            urls:
              - "/"
            children: []
          - name: "bundles/product"
            urls:
              - "/spodnie-1"
            children: []
          - name: "bundles/cart"
            urls:
              - "/checkout/cart"
              - "/checkout/cart/"
            children: []
          - name: "customer-account"
            urls:
              - "customer/account/login/"
              - "/customer/account/index/"
              - "/customer/account/index"
              - "/customer/account/edit"
              - "/customer/account/edit/"
              - "sales/order/view/order_id/121/"
              - "/customer/address/new/"
              - "/sales/order/history/"
              - "/aw_rma/customer/"
              - "/wishlist"
            children: []
          - name: "club"
            urls:
              - "/club"
            children: []
  - name: "bundles/checkout"
    urls:
      - "/checkout"
      - "/checkout/"
      - "/checkout/#payment"
    children: []

build.json file is empty

Hey,
Using the default config in the README, I get an empty build.json file.

I might being a bit too keen, or if I have not followed the instructions correctly - I know you are going to add some demo code.

Thanks!

I have a file public_html/config-gen.yml:

presets:
  - name: m2
    options:
      bundle_config: file:test/fixtures/bundle-config.yml

And public_html/test/fixtures/bundle-config.yml:

bundles:
  - name: "bundles/main"
    urls:
      - "/"
    children:
      - name: "bundles/product"
        urls:
          - "/test-bracelet/"
        children: []

But when I try and visit (after visiting https://127.0.0.1:8080/ once) https://127.0.0.1:8080/__bs/build.json gives me this:

{
  "generateSourceMaps": true,
  "inline_text": true,
  "optimize": "none",
  "deps": [],
  "map": {},
  "config": {},
  "shim": {},
  "paths": {},
  "modules": [
    {
      "name": "requirejs/require",
      "include": [],
      "exclude": [],
      "create": false
    },
    {
      "name": "bundles/main",
      "include": [],
      "exclude": [
        "requirejs/require"
      ],
      "create": true
    },
    {
      "name": "bundles/product",
      "include": [],
      "exclude": [
        "requirejs/require",
        "bundles/main"
      ],
      "create": true
    }
  ]
}