Code Monkey home page Code Monkey logo

webworkify-webpack's Introduction

webworkify-webpack

Generates a web worker at runtime from webpack's bundled modules with only the used dependencies. Possible because of webpack's module structure. Just require.resolve(PATH_TO_MODULE) the module you want to be the worker's entry point.

inspired by webworkify

install

npm install webworkify-webpack --save

v2 vs v1

For v1 go to: 1.1.8

Version 2 uses webpack's api as much as possible vs the hacky implementation of version 1 (I wasn't aware of webpack's api while writing it) which did the job but with some inconsistency between different browsers and some caveats.

In v2:

  • no limitation on webpack's devtool - eval was forbidden in v1.
  • no issues with anonymous functions exported as module.exports - there were issues with anonymous functions in v1.
  • require.resolve instead of regular require\import - The only limitation is using require.resolve which means that currently the code using webworkify-webpack is coupled to the build tool (webpack - but who uses webworkify-webpack already uses webpack) and its not possible to use es2015 modules => checkout out the future work section.

webworkify-webpack vs webpack's worker-loader and target: 'webworker'

webworkify-webpack allows to use one bundle for running same code both on browser and web worker environments. webpack's current alternatives for web workers are creating bundle which can be run in a web worker environment only and can results in 2 separate files like in the worker-loader case (one file for browser and one for web worker => code duplication).

The motivation for webworkify-webpack was creating a library which expose to the user the same functionality both in sync and async forms. I wanted to keep one bundle in order to reduce complexity of using external library to the minimum and make bundle size as minimal as possible when using external library which supports both sync and async functionality (without code duplication).

Since webpack's solutions for web workers are being constructed at compile time, the added value is that its possible to use dev tools like hmr (at least when using target: 'webworker') which isn't possible with webworkify-webpack.
In addition, regular js syntax is being used without the need to use require.resolve as in the webworkify-webpack case => checkout out the future work section.

methods

import work from 'webworkify-webpack'

let w = work(require.resolve(modulePath) [, options])

Return a new web worker from the module at modulePath.

The file at modulePath should export its worker code in module.exports as a function that will be run with no arguments.

Note that all the code outside of the module.exports function will be run in the main thread too so don't put any computationally intensive code in that part. It is necessary for the main code to require() the worker code to fetch the module reference and load modulePath's dependency graph into the bundle output.

options

  • all - bundle all the dependencies in the web worker and not only the used ones. can be useful in edge cases that I'm not aware of when the used dependencies aren't being resolved as expected due to the runtime regex checking mechanism or just to avoid additional work at runtime to traverse the dependencies tree.
  • bare - the return value will be the blob constructed with the worker's code and not the web worker itself.

example

First, a main.js file will launch the worker.js and print its output:

import work from 'webworkify-webpack';

let w = work(require.resolve('./worker.js'));
w.addEventListener('message', event => {
    console.log(event.data);
});

w.postMessage(4); // send the worker a message

then worker.js can require() modules of its own. The worker function lives inside of the module.exports:

import gamma from 'gamma'

module.exports = function worker (self) {
    self.addEventListener('message', (event) => {
        const startNum = parseInt(event.data); // ev.data=4 from main.js
        setInterval(() => {
            const r = startNum / Math.random() - 1;
            self.postMessage([ startNum, r, gamma(r) ]);
        }, 500);
    });
};

Now after webpackifying this example, the console will contain output from the worker:

[ 4, 0.09162078520553618, 10.421030346237066 ]
[ 4, 2.026562457360466, 1.011522336481017 ]
[ 4, 3.1853125018703716, 2.3887589540750214 ]
[ 4, 5.6989969260510005, 72.40768854476167 ]
[ 4, 8.679491643020487, 20427.19357947782 ]
[ 4, 0.8528139834191428, 1.1098187157762498 ]
[ 4, 8.068322137547542, 5785.928308309402 ]
...

future work

The goal is to make webworkify-webpack fully based on webpack's api. I'm not sure how to accomplish it since I never wrote a webpack loader\plugin (is it possible other way?) so I'm asking for help :)

Points of view:

  1. webpackBootstrapFunc - should be taken from webpack's source.
  2. ability to use regular module import\require (not require.resolve) but still passing the module id to 'webworkify-webpack'.
  3. ability to know all specific's module dependencies in compile time so there is no need to traverse the dependencies tree in runtime with regular expressions (when uglifying the code the web worker's bundle can include more dependencies than only the used ones because regular expressions nature).
  4. if there is going to be build in compile time, what about hmr as dev tool ?
  5. is the ability 'webworkify-webpack' provides should be part of webpack core as another form of web workers support or should it remain as external module ?

license

MIT

webworkify-webpack's People

Contributors

amilajack avatar austinhyde avatar borisirota avatar d1manson avatar extronics avatar janicduplessis avatar lafin avatar taylor1791 avatar zuchaowang avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar

webworkify-webpack's Issues

Webworkify-webpack v2.1.5 config for node_modules dependency `Cannot read property 'call' of undefined`

Hi,
I can't get webworkify-webpack working for a node_module dependency that uses webworkify v^1.2.1 and uses the require('./worker') rather than require.resolve('./worker') for v2.1.5
Here is my webpack.config.js.

module.exports = {
  entry: {
    app: ['./example/main.js']
  },
  output: {
    filename: 'output.js',
    pathinfo: true
  },
  module: {
    rules: [
      {
        test: /\.jsx?$/,
        exclude: /node_modules/,
        use: [
          {
            loader: 'babel-loader',
            options: {
              presets: ['es2015']
            }
          }
        ]
      }
    ]
  },
  resolve: {
      alias: {
          'webworkify': 'webworkify-webpack'
      }
  },
  plugins: [
    new webpack.NamedModulesPlugin()
  ]
}

How can i get it working with my dependency without having to alter my dependency code? Thanks, your help is greatly appreciated!

Spawning child workers

I'm having some trouble getting workers created through webworkify-webpack to spawn child workers. Is this supported functionality? Alternatively, am I being dumb: Inside of my initial worker i have code:

import work from 'webworkify-webpack';
const aggWorker = work(require.resolve('./agg-worker.js'));

Which then gives a "Uncaught ReferenceError: window is not defined" error in chrome. Thanks

Webpack 5 support?

I recently noticed when upgrading my app to Webpack 5 that the blob webworkify-webpack was generating was significantly different than the one being created using Webpack 4. It seems like not all dependencies that were once included are being generated.

Are there plans to support Webpack 5?

note: just adding the all option did not impact my end state. While more was included, it did not appear to be everything.

Content-Length: 7306 // Webpack 5

Content-Length: 227661 // Webpack 4

webpack 4 support

Looks like this isn't compatible with webpack 4 which is already in beta. Do you plan to support it?

This library is awesome

The webpack worker loader is extremely confusing to set up, and on top of that the examples are for webpack 2 only and aren't backwards compatible for webpack 1 and don't mention this fact anywhere.

It could just be I find webpack loaders confusing in general but this is much easier.

It must work by creating an inline web worker, correct? Since it doesn't require you to make an extra script tag or anything.

Jest test

Hi, I absolutely love this package, but is there any way to test my code in jest when I use webworkify-webpack?

I'm getting this error

ReferenceError: __webpack_modules__ is not defined

__webpack_modules__ is not defined

Hi there, I've researched for similar error but could not find answer so I decided to post it here.

Does anybody know what could be a problem? Because same setup worked until I merged something and that didn't actually modified place where this error happened.

main: __webpack_modules__
          ^

ReferenceError: __webpack_modules__ is not defined
    at module.exports (/Users/vhmac/Developer/abago/node_modules/webworkify-webpack/index.js:154:11)
    at Object.<anonymous> (/Users/vhmac/Developer/abago/src/store/index.js:21:16)
    at Module._compile (module.js:571:32)
    at loader (/usr/local/lib/node_modules/babel-cli/node_modules/babel-register/lib/node.js:144:5)
    at Object.require.extensions.(anonymous function) [as .js] (/usr/local/lib/node_modules/babel-cli/node_modules/babel-register/lib/node.js:154:7)
    at Module.load (module.js:488:32)
    at tryModuleLoad (module.js:447:12)
    at Function.Module._load (module.js:439:3)
    at Module.require (module.js:498:17)
    at require (internal/module.js:20:19)```

String matching for functions.

7996812

string matching here won't work if the worker file defines its export function without a space, since the toString will always inject a space. Will happen very often with code minifiers.
(function(e){}).toString() // produces "function (e){}"

Cannot be used as a webworkify drop in anymore.

With 2.0.0 the interface of this library is diverged from https://github.com/substack/webworkify . Therefore it is no longer possible to use webworkify-wbpack as a direct alternative to webworkify via:

resolve: {
    alias: {
        webworkify: 'webworkify-webpack'
    }
}

Are there any plans to fix it? I was trying to make 1.1.8 work as a workify drop in with webpack 2 at https://github.com/cansin/webworkify-webpack2 , but having hard time figuring out what's going wrong with uglify-js atm.

v2 - self not being passed to webworker?

Trying to get this working with react-map-gl.

Here is their code:

var workerURL = window.URL.createObjectURL(new WebWorkify(require('../../source/worker'), {bare: true}));

For some reason this fails when they construct the worker like so:

new window.Worker(workerURL);

The Worker constructor (https://github.com/mapbox/mapbox-gl-js/blob/master/js/source/worker.js) expects a constructor parameter that has an addEventListener function, but is being passed a string ENTRY_MODULE so it blows up.

Any ideas?

Uncaught TypeError: (intermediate value) ... is not a function

When I use these codes like you explaininig, the process is normal, but it will alwasy throw a error:

Uncaught TypeError: (intermediate value)(intermediate value)(intermediate value)(intermediate value)(intermediate value)(intermediate value)(intermediate value)(intermediate value)(intermediate value)(intermediate value)(intermediate value)(...) is not a function

and, my codes are:

this._worker = worker(require.resolve('./chunks.worker'));

the fakeCode of chunks.worker.js is:

import xx from "xxx";

class WorkerController{
    // ...
}

new WorkerController;

v3

https://webpack.js.org/api/plugins/
https://webpack.js.org/development/how-to-write-a-plugin/
https://webpack.js.org/development/plugin-patterns/
https://github.com/webpack/docs/wiki/how-to-write-a-plugin
https://webpack.js.org/api/plugins/compiler/
https://webpack.js.org/api/plugins/compilation/
https://webpack.js.org/api/parser/
https://github.com/webpack-contrib/worker-loader
https://github.com/webpack/webpack/tree/master/lib/webworker
https://www.youtube.com/watch?v=NHI_PhoykVU
https://github.com/TheLarkInn/everything-is-a-plugin
https://github.com/TheLarkInn/artsy-webpack-tour

https://medium.com/webpack/contributors-guide/home

extract-text-webpack-plugin:
https://github.com/webpack-contrib/extract-text-webpack-plugin/blob/master/src/index.js
https://github.com/webpack-contrib/extract-text-webpack-plugin/blob/master/src/loader.js

Preact async loader:

  1. https://github.com/developit/preact-cli/blob/master/src/lib/webpack/async-component-loader.js
  2. https://github.com/developit/preact-cli/blob/master/src/components/async.js

it isn't possible to create new files in loader so there is need to make a plugin if I want to make one worker builder module:
https://webpack.js.org/api/loaders/
webpack/webpack#3311 (comment)
webpack/webpack.js.org#116

Maybe the best solution is to make a plugin that:

  1. generates a webworkify module once (based on https://github.com/webpack/webpack/blob/master/lib/MainTemplate.js) and adds it as a module (where ? to which chunk ? how can I know to where ?). Maybe this should be done by a loader ? its possible if we still import and call the webworkify-webpack module directly + it can be used as a drop in replacement for webworkify
  2. for every worker file, generate a wrapper which gets the module dependencies (including dll) of the module, replace the dependency of the original module and this way injects its result into webworkify-webpack.

https://github.com/jamiehill/path-override-webpack-plugin
https://github.com/webpack/webpack/blob/master/lib/NormalModuleFactory.js#L218
https://github.com/webpack/webpack/blob/master/lib/FunctionModuleTemplatePlugin.js

DLL:
https://github.com/webpack/webpack/blob/master/lib/DllReferencePlugin.js
https://github.com/webpack/webpack/blob/master/lib/DelegatedModuleFactoryPlugin.js
https://github.com/webpack/webpack/blob/master/lib/DelegatedModule.js

Erroneous detection of dependencies can lead to syntax error in Worker

This library attempts to find dependencies using a regular expression:

// main bundle deps
var re = new RegExp('(\\\\n|\\W)' + quoteRegExp(webpackRequireName) + dependencyRegExp, 'g')
var match
while ((match = re.exec(fnString))) {
if (match[3] === 'dll-reference') continue
retval[queueName].push(match[3])
}

In minified code, webpackRequireName will often be a single-character function name like r, so this regex will detect all occurrences of r(…). Unfortuantely, the r() function name may also be used in other scopes for a totally different function, so this dependency detection can pick up a totally random argument and treat it as a dependency.

In our case (as users of hls.min.js) this function call is minified to r('init', null), and so 'init' is erroneously detected as a dependency.

Ordinarily this fails silently and doesn't affect functionality. When assembling the WebWorker blob, for id='init', the call to sources.main[id] returns undefined and things carry on without raising any errors. However, if there is variable/function defined on the Array class with the name init, then it will be returned, stringified, and lead to a syntax error in the Worker. (In our case, that init function is added to the Array prototype by ember.js)

So, with both of those very unlucky circumstances, we end up with this when the Worker is initialized:

Uncaught SyntaxError: Unexpected token '{' (at efc643e5-eb1d-4408-939d-225e166586ac:135:20)

I imagine a robust fix to the dependency-detection will be tricky - parsing JS with Regex is naturally not going to be perfect. But I wonder if it might be possible to mitigate the errors when it does happen 🤔

Provide documentation details on difference between this and webpack's target: 'webworker'

I was wondering if it would be possible that you provide some information on how using this vs webpack's built in target: 'webworker' differs.

It could provide users with multiple options or justifications for using one over the other. Plus if some of the things this library is trying to accomplish can be done through the webpack target: 'webworker', then we should get a PR in for the sake of updating that section of our core codebase!! Excellent work and thank you for the contributions to the webpack community.

TypeError: sources.map is not a function

First of all, thanks for a great lib, really saved me some stress about releasing a feature im working on.

So, i have a webpack.config.js with a few entry points, i tried to import webworkify-webpack in one of them (along with other imports) and use it for a heavily computational task, this didnt work by default (threw same error message as in title) so i tried adding another entry point to my webpack config that would only import this library just to see if it might work then, which it did.

However, i found out after some time that it didnt have to do with whether i imported the lib or not, but with whether sources (https://github.com/borisirota/webworkify-webpack/blob/master/index.js#L2) is an array or not, and by adding this extra empty entry point, sources became an array..

I have no idea why that happens but maybe you have a clearer understanding of this?

Webpack config without empty test entrypoint:
screen shot 2016-03-24 at 4 23 39 pm
screen shot 2016-03-24 at 4 23 05 pm

Webpack config with empty test entrypoint:
screen shot 2016-03-24 at 4 25 29 pm
screen shot 2016-03-24 at 4 22 24 pm

Doesn't work in ie10

When creating a web worker in ie10, the page fails to load and in the console ie10 throws a “SCRIPT5002 function expected”

Problems with the import in webpack 4

Hello. I have been using your awesome library for a while. Recently updated my project to webpack "4.29.5", after that, all the workers started to complain: Uncaught TypeError: webpack_require.t is not a function. This error appears as soon as there is an import or a require statement in the worker. Any ideas how to fix this?

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.