developit / workerize Goto Github PK

View Code? Open in Web Editor NEW

4.3K 51.0 91.0 64 KB

🏗️ Run a module in a Web Worker.

Home Page: https://github.com/developit/workerize-loader

JavaScript 86.90% HTML 13.10%

web-worker web-workers threading

workerize's Introduction

💖 Using Webpack? You want workerize-loader ➡️

Workerize

Moves a module into a Web Worker, automatically reflecting exported functions as asynchronous proxies.

Bundles a tiny, purpose-built RPC implementation into your app
If exported module methods are already async, signature is unchanged
Supports synchronous and asynchronous worker functions
Works beautifully with async/await
Just 800 bytes of gzipped ES3

Install

npm install --save workerize

Usage

Pass either a function or a string containing code.

worker.js:

let worker = workerize(`
	export function add(a, b) {
		// block for half a second to demonstrate asynchronicity
		let start = Date.now();
		while (Date.now()-start < 500);
		return a + b;
	}
`);

(async () => {
	console.log('3 + 9 = ', await worker.add(3, 9));
	console.log('1 + 2 = ', await worker.add(1, 2));
})();

License

MIT License © Jason Miller

workerize's People

Contributors

Stargazers

Watchers

Forkers

richieahb aminbenselim cbforks derickwarshaw cades whjvenyl linecode kakahanhaoniao gi972 esmevane juanjoseva karamata lris karol-majewski nickli libin1991 lp249839965 dnakov starhai2724 hhy5277 mengyou658 kdncode madmax983 dudarau volkancakil josesaribeiro fanker killsos articlabs sauravazad pastrylabs gregorystarr iamsingularity happy-ferret ykankaya l-chris boblauer vishaljoshi jamtis rainstormza anima-os-stash yuebin cuulee shepardzhao ulissepress tkien indigos33k3r wegiangb odeds the-cc-dev b3838438org fengweijp hero adwardlzq pea-jj breezefeng devcodo brixdan brodybits forkkit codinronan xinnianyoulin bozvul993 regexp-lin johnsonjo4531 thephoenixofthevoid rajeshkhetarpal cloudyjeep lapd-frontend champi-dev doc22940 satyamkumaronline shakahl cybernetics thekevinscott breakinferno alireza-mh jbro885 ksmore mzoupas uiforks 00mjk hesitanting nhkyxx rake7h fruitbox12 nikonee ydyvip antespace zhy15608103017 charliekaks

workerize's Issues

Feature: worker pooling

Hello,

It would be great if I could create a pool of workers. That way, parallelizing hot paths could be completely abstracted using workerize.

Considerations:

To implement worker pooling, a worker may have to know when another worker is busy, so it can handle take over handling new requests until becomes busy itself
Since workers independently manage memory, they would all have to run the same code, and there would have to be a way to run a function on all workers simultaneously in order to share state

It will be interesting to see what we will need to do for this. Would you accept a PR?

Thanks for considering!

default export fails to build with create-react-app

When importing workerize in a cra app the build fail with this message

> react-scripts build

Creating an optimized production build...
Failed to compile.

Failed to minify the code from this file:

 	./node_modules/workerize/src/index.js:23

Read more here: http://bit.ly/2tRViJ9

I was able to make it work using import workerize from 'workerize/dist/workerize.m.js'

External deps

We very often want to use libraries like lodash in compute intensive routines that would be good to run in workers. Have you thought about how to solve this problem?

Support pass OffscreenCanvas object.

const worker = workerize(`
        export function drawImage(canvas,image){ return canvas.getContext("2d").drawImage(image, 0, 0)}
         `)(canvas, image)
worker.drawImage(document.querySelector('canvas'). transferControlToOffscreen(),img)

it will throw

DOMException: Failed to execute 'postMessage' on 'Worker': An OffscreenCanvas could not be cloned because it was not transferred.

https://developer.mozilla.org/en-US/docs/Web/API/Transferable

Consider removing expose(), call() and kill() ?

	worker.expose = methodName => {
		worker[i] = function() {
			return worker.call(methodName, [].slice.call(arguments));
		};
	};

Instead methodName parameter should be used:

	worker.expose = methodName => {
		worker[methodName] = function() {
			return worker.call(methodName, [].slice.call(arguments));
		};
	};

Beside that I don't see why the call and expose methods are accessible from outside. (Don't assign to worker at all)
The purpose of the kill method is also not clear for me.

Whether the thread pool idea was introduced?

Whether the thread pool idea was introduced?The thread pool is a good idea to Improve efficiency

Handling no Worker support

🤔 Based on my first few seconds with workerize and workerize-loader:

In the situations where Workers aren't supported (IE9, Chrome w/ file:/// protocol, etc), the Worker construction fails and throws. This would force the user to try/catch the worker construction and then implement code to run in the main thread in that case (if necessary).

Rather that try/catching that and having to write that fall-back code, it could be handy if workerize could fail gracefully and just proxy through the original functions (wrapped in Promises to preserve the expected API).

Thoughts? I don't have a lot of experience with Workers / workerize, but it came to my mind so I thought I'd bring it up.

Edit: Forgot to mention that workerize and the loader look super cool so far, so great work! 👍

Add support for async generator functions

Promises are great, but async iterators would be even more general.

non-enumerable object properties are lost after passing an object to workerize.

Here's a real life example:

// The main thread
import Worker from 'workerize-loader!./workers';
import getCompaniesFromFirebase from './getCompaniesFromFirebase';

const worker = Worker();

const {
  filterCompaniesByTypes
  // Other modules
} = worker;

(async () => {
  const companies = await getCompaniesFromFirebase();
  // typeof companies[0].id === "string" // true

  const companiesFiltered = await filterCompaniesByTypes(companies, ['company-type-1', 'company-type-2']);
  // typeof companiesFiltered[0].id === "string" // false
  // typeof companiesFiltered[0].id === "undefined" // true
})();

// The worker
import intersection from 'lodash/intersection';

export const filterCompaniesByTypes = (companies, typesToFilerBy) => {
  return companies.filter(({ types, id }) => {
    // typeof id === 'undefined' // true
    return intersection(types, typesToFilerBy).length > 0;
  });
};

In the given example the getCompaniesFromFirebase() function returns an array of objects. Each object has some properties, inluding the id property that is non-enumerable.

After executing the filterCompaniesByTypes() function (which is a web worker function) I receive a filtered array. All id properties are gone.

if it's the desired behaviour (or an unpleasant, but expected, side-effect) not a bug, then it would be good to mention it in the readme.

An in-range update of microbundle is breaking the build 🚨

Version 0.4.2 of microbundle was just published.

Branch	Build failing 🚨
Dependency	microbundle
Current Version	0.4.1
Type	devDependency

This version is covered by your current version range and after updating it in your project the build failed.

microbundle is a devDependency of this project. It might not break your production code or affect downstream projects, but probably breaks your build or test tools, which may prevent deploying or publishing.

Status Details

❌ continuous-integration/travis-ci/push The Travis CI build failed Details

Commits

The new version differs by 7 commits.

096b8ba 0.4.2
fa91a22 Add JSX test
b38b860 Fix Nodent usage of acorn-jsx. (thanks @songawee - see developit/karmatic#8)
a133de8 Show error stack traces
004adc6 Merge pull request #59 from Andarist/snapshots
e8cb46f Merge branch 'master' into snapshots
c2627c7 Auto generate tests with snapshots

See the full diff

FAQ and help

There is a collection of frequently asked questions. If those don’t help, you can always ask the humans behind Greenkeeper.

Your Greenkeeper Bot 🌴

React native support?

Hi, wanted to check if there are any difficulties when using workerize with react native? There are solutions that achieve it i.e https://github.com/austinksmith/Hamsters.js But as always your api is much cleaner and minimal.

universal-ify the pre processing of workerize-loader?

This is more a discussion than an issue.

I'm building a worker-map library that takes a loop and splits it into several workers (following the map/reduce pattern). The downside right now is that all the worker code has to be passed as a string (because as @developit already said in a tweet export isn't valid syntax in a closure)

Here's the API at the moment.

import workerize from 'workerize'

async function workerMap(consumerString, data, {concurrency = 4} = {}) {
  const promisesArr = []
  const step = Math.ceil(data.length / concurrency)

  // map
  for (let i = 0; i < data.length; i += step) {
    const consumer = workerize(consumerString)
    // expecting a export consume(array, workerNb) function in the worker
    promisesArr.push(consumer.consume(data.slice(i, i + step), i / step + 1))
  }
  const ret = await Promise.all(promisesArr)

  // reduce
  const concated = ret.reduce((acc, cur) => acc.concat(cur), [])

  return concated
}

What do you think would be doable/a good idea to do? Is it even possible to extract the code from a module import?

Quentin

Integration with Rollup

Hey, I'm curious if it would be possible to workerize web workers in rollup? Do you maybe have a plan to support rollup using a plugin?

Readme demo says 250ms is "half a second"

..should be "a quarter second".

This is the code:

// block for half a second to demonstrate asynchronicity
let start = Date.now();
while (Date.now()-start < 250);

An in-range update of microbundle is breaking the build 🚨

Version 0.4.3 of microbundle was just published.

Branch	Build failing 🚨
Dependency	microbundle
Current Version	0.4.2
Type	devDependency

This version is covered by your current version range and after updating it in your project the build failed.

Status Details

❌ continuous-integration/travis-ci/push The Travis CI build failed Details

Commits

The new version differs by 11 commits.

6ae3969 0.4.3
ad72047 Update snapshots with smaller code size
b5504c9 Merge pull request #69 from tymondesigns/feature/newer-typescript
edf2975 Merge branch 'master' into feature/newer-typescript
2413292 Update deps
cc87186 Move to uglify-es and rollup-plugin-uglify@3!
6a06939 Update deps
d671cf8 Quickfix for rollup-plugin-nodent breaking JSX plugin. (see oligot/rollup-plugin-nodent#6)
a1d926f use latest typescript and fix issue with importHelpers
99746d8 Merge pull request #65 from tlvince/64
f9b6e02 Guard against flow transform response

See the full diff

FAQ and help

There is a collection of frequently asked questions. If those don’t help, you can always ask the humans behind Greenkeeper.

Your Greenkeeper Bot 🌴

An in-range update of eslint is breaking the build 🚨

Version 4.17.0 of eslint was just published.

Branch	Build failing 🚨
Dependency	eslint
Current Version	4.16.0
Type	devDependency

This version is covered by your current version range and after updating it in your project the build failed.

eslint is a devDependency of this project. It might not break your production code or affect downstream projects, but probably breaks your build or test tools, which may prevent deploying or publishing.

Status Details

❌ continuous-integration/travis-ci/push The Travis CI build failed Details

Commits

The new version differs by 11 commits.

2af9446 4.17.0
5ad3fb2 Build: changelog update for 4.17.0
1da1ada Update: Add "multiline" type to padding-line-between-statements (#8668)
bb213dc Chore: Use messageIds in some of the core rules (#9648)
1aa1970 Docs: remove outdated rule naming convention (#9925)
3afaff6 Docs: Add prefer-destructuring variable reassignment example (#9873)
d20f6b4 Fix: Typo in error message when running npm (#9866)
51ec6a7 Docs: Use GitHub Multiple PR/Issue templates (#9911)
dc80487 Update: space-unary-ops uses astUtils.canTokensBeAdjacent (fixes #9907) (#9906)
084351b Docs: Fix the messageId example (fixes #9889) (#9892)
9cbb487 Docs: Mention the globals key in the no-undef docs (#9867)

See the full diff

FAQ and help

There is a collection of frequently asked questions. If those don’t help, you can always ask the humans behind Greenkeeper.

Your Greenkeeper Bot 🌴

Using in Parcel bundler

Parcel is another very popular alternative to Webpack.

Arrow functions

would it be possible to use arrow functions to workerize?

An in-range update of eslint is breaking the build 🚨

Version 4.16.0 of eslint was just published.

Branch	Build failing 🚨
Dependency	eslint
Current Version	4.15.0
Type	devDependency

This version is covered by your current version range and after updating it in your project the build failed.

Status Details

❌ continuous-integration/travis-ci/push The Travis CI build failed Details

Release Notes

v4.16.0

e26a25f Update: allow continue instead of if wrap in guard-for-in (fixes #7567) (#9796) (Michael Ficarra)
af043eb Update: Add NewExpression support to comma-style (#9591) (Frazer McLean)
4f898c7 Build: Fix JSDoc syntax errors (#9813) (Matija Marohnić)
13bcf3c Fix: Removing curly quotes in no-eq-null report message (#9852) (Kevin Partington)
b96fb31 Docs: configuration hierarchy for CLIEngine options (fixes #9526) (#9855) (PiIsFour)
8ccbdda Docs: Clarify that -c configs merge with .eslintrc.* (fixes #9535) (#9847) (Kevin Partington)
978574f Docs: Fix examples for no-useless-escape (#9853) (Toru Kobayashi)
cd5681d Chore: Deactivate consistent-docs-url in internal rules folder (#9815) (Kevin Partington)
2e87ddd Docs: Sync messageId examples' style with other examples (#9816) (Kevin Partington)
1d61930 Update: use doctrine range information in valid-jsdoc (#9831) (Teddy Katz)
133336e Update: fix indent behavior on template literal arguments (fixes #9061) (#9820) (Teddy Katz)
ea1b15d Fix: avoid crashing on malformed configuration comments (fixes #9373) (#9819) (Teddy Katz)
add1e70 Update: fix indent bug on comments in ternary expressions (fixes #9729) (#9818) (Teddy Katz)
6a5cd32 Fix: prefer-destructuring error with computed properties (fixes #9784) (#9817) (Teddy Katz)
601f851 Docs: Minor modification to code comments for clarity (#9821) (rgovind92)
b9da067 Docs: fix misleading info about RuleTester column numbers (#9830) (Teddy Katz)
2cf4522 Update: Rename and deprecate object-property-newline option (#9570) (Jonathan Pool)
acde640 Docs: Add ES 2018 to Configuring ESLint (#9829) (Kai Cataldo)
ccfce15 Docs: Minor tweaks to working with rules page (#9824) (Kevin Partington)
54b329a Docs: fix substitution of {{ name }} (#9822) (Andres Kalle)

Commits

The new version differs by 22 commits.

33ca1ea 4.16.0
1a9ddee Build: changelog update for 4.16.0
e26a25f Update: allow continue instead of if wrap in guard-for-in (fixes #7567) (#9796)
af043eb Update: Add NewExpression support to comma-style (#9591)
4f898c7 Build: Fix JSDoc syntax errors (#9813)
13bcf3c Fix: Removing curly quotes in no-eq-null report message (#9852)
b96fb31 Docs: configuration hierarchy for CLIEngine options (fixes #9526) (#9855)
8ccbdda Docs: Clarify that -c configs merge with .eslintrc.* (fixes #9535) (#9847)
978574f Docs: Fix examples for no-useless-escape (#9853)
cd5681d Chore: Deactivate consistent-docs-url in internal rules folder (#9815)
2e87ddd Docs: Sync messageId examples' style with other examples (#9816)
1d61930 Update: use doctrine range information in valid-jsdoc (#9831)
133336e Update: fix indent behavior on template literal arguments (fixes #9061) (#9820)
ea1b15d Fix: avoid crashing on malformed configuration comments (fixes #9373) (#9819)
add1e70 Update: fix indent bug on comments in ternary expressions (fixes #9729) (#9818)

There are 22 commits in total.

See the full diff

FAQ and help

There is a collection of frequently asked questions. If those don’t help, you can always ask the humans behind Greenkeeper.

Your Greenkeeper Bot 🌴

Accept a module path instead of code string?

It seems this library current accepts code string (and somehow tries to convert it to CommonJS 🤔) but can it accept a module path? I guess it should be possible to generate a code to import the module from the given path.

An in-range update of eslint is breaking the build 🚨

☝️ Greenkeeper’s updated Terms of Service will come into effect on April 6th, 2018.

Version 4.19.0 of eslint was just published.

Branch	Build failing 🚨
Dependency	eslint
Current Version	4.18.2
Type	devDependency

This version is covered by your current version range and after updating it in your project the build failed.

Status Details

❌ continuous-integration/travis-ci/push The Travis CI build failed Details

Release Notes

v4.19.0

55a1593 Update: consecutive option for one-var (fixes #4680) (#9994) (薛定谔的猫)
8d3814e Fix: false positive about ES2018 RegExp enhancements (fixes #9893) (#10062) (Toru Nagashima)
935f4e4 Docs: Clarify default ignoring of node_modules (#10092) (Matijs Brinkhuis)
72ed3db Docs: Wrap Buffer() in backticks in no-buffer-constructor rule description (#10084) (Stephen Edgar)
3aded2f Docs: Fix lodash typos, make spacing consistent (#10073) (Josh Smith)
e33bb64 Chore: enable no-param-reassign on ESLint codebase (#10065) (Teddy Katz)
66a1e9a Docs: fix possible typo (#10060) (Vse Mozhet Byt)
2e68be6 Update: give a node at least the indentation of its parent (fixes #9995) (#10054) (Teddy Katz)
72ca5b3 Update: Correctly indent JSXText with trailing linebreaks (fixes #9878) (#10055) (Teddy Katz)
2a4c838 Docs: Update ECMAScript versions in FAQ (#10047) (alberto)

Commits

The new version differs by 12 commits.

4f595e8 4.19.0
16fc59e Build: changelog update for 4.19.0
55a1593 Update: consecutive option for one-var (fixes #4680) (#9994)
8d3814e Fix: false positive about ES2018 RegExp enhancements (fixes #9893) (#10062)
935f4e4 Docs: Clarify default ignoring of node_modules (#10092)
72ed3db Docs: Wrap Buffer() in backticks in no-buffer-constructor rule description (#10084)
3aded2f Docs: Fix lodash typos, make spacing consistent (#10073)
e33bb64 Chore: enable no-param-reassign on ESLint codebase (#10065)
66a1e9a Docs: fix possible typo (#10060)
2e68be6 Update: give a node at least the indentation of its parent (fixes #9995) (#10054)
72ca5b3 Update: Correctly indent JSXText with trailing linebreaks (fixes #9878) (#10055)
2a4c838 Docs: Update ECMAScript versions in FAQ (#10047)

See the full diff

FAQ and help

There is a collection of frequently asked questions. If those don’t help, you can always ask the humans behind Greenkeeper.

Your Greenkeeper Bot 🌴

Try async-proxy module

This project is very similar to my worker-sandbox, so I think you probably can use async-proxy module to make the code look clearer.

An in-range update of microbundle is breaking the build 🚨

Version 0.3.1 of microbundle was just published.

Branch	Build failing 🚨
Dependency	microbundle
Current Version	0.3.0
Type	devDependency

This version is covered by your current version range and after updating it in your project the build failed.

Status Details

❌ continuous-integration/travis-ci/push The Travis CI build failed Details

Commits

The new version differs by 3 commits.

932acd6 0.3.1
d10fb70 second demo
8960114 If defaults + named exports are detected, only export default and prune named. Enable pure properties for tree-shaking (since they're already enabled for uglify)

See the full diff

FAQ and help

There is a collection of frequently asked questions. If those don’t help, you can always ask the humans behind Greenkeeper.

Your Greenkeeper Bot 🌴

How to pass a function

Usage
Pass either a function or a string containing code.

Could you please show in the example how to pass a function?

An in-range update of microbundle is breaking the build 🚨

Version 0.4.4 of microbundle was just published.

Branch	Build failing 🚨
Dependency	microbundle
Current Version	0.4.3
Type	devDependency

This version is covered by your current version range and after updating it in your project the build failed.

Status Details

❌ continuous-integration/travis-ci/push The Travis CI build failed Details

Commits

The new version differs by 27 commits.

b20b79b 0.4.4
8905cd4 Merge pull request #71 from freeman/rollup-plugin-typescript2
87d7051 Merge pull request #79 from cristianbote/source-map-option
81c8e4c Merge pull request #82 from Andarist/patch-1
b8ba766 Merge pull request #84 from Andarist/code-of-conduct
07bf0b9 Add CODE_OF_CONDUCT.md
4c7182b Fixed crash when bundling packages without dependencies
fdf9034 Update readme with --sourcemap option
b4dcc27 Drop the undefined check and just use the ! check
67b7c52 Added better value checking
9cd9a06 Ammend the test to reflect the defaults
845c6c5 Added source-map option
7a570c8 Merge pull request #78 from tusbar/update-nodent
a3440d4 Use upstream rollup-plugin-nodent
1bd22b7 upgrade to [email protected]

There are 27 commits in total.

See the full diff

FAQ and help

There is a collection of frequently asked questions. If those don’t help, you can always ask the humans behind Greenkeeper.

Your Greenkeeper Bot 🌴

developit / workerize Goto Github PK

workerize's Introduction

💖 Using Webpack? You want workerize-loader ➡️

Workerize

Install

Usage

License

workerize's People

Contributors

Stargazers

Watchers

Forkers

workerize's Issues

Version 0.4.2 of microbundle was just published.

Version 0.4.3 of microbundle was just published.

Version 4.17.0 of eslint was just published.

Version 4.16.0 of eslint was just published.

Version 4.19.0 of eslint was just published.

Version 0.3.1 of microbundle was just published.

Version 0.4.4 of microbundle was just published.

Recommend Projects

Recommend Topics

Recommend Org