Code Monkey home page Code Monkey logo

throat's Introduction

throat

Throttle the parallelism of an asynchronous, promise returning, function / functions. This has special utility when you set the concurrency to 1. That way you get a mutually exclusive lock.

Professionally supported throat is now available

Build Status Coveralls github branch Rolling Versions NPM version

Installation

npm install throat

API

throat(concurrency)

This returns a function that acts a bit like a lock (exactly as a lock if concurrency is 1).

Example, only 2 of the following functions will execute at any one time:

const throat = require('throat')(2);

const resA = throat(async () => {
  /* async stuff... */
});
const resB = throat(async () => {
  /* async stuff... */
});
const resC = throat(async () => {
  /* async stuff... */
});
const resD = throat(async () => {
  /* async stuff... */
});
const resE = throat(async () => {
  /* async stuff... */
});

throat(concurrency, worker)

This returns a function that is an exact copy of worker except that it will only execute up to concurrency times in parallel before further requests are queued:

const throat = require('throat');

const input = ['fileA.txt', 'fileB.txt', 'fileC.txt', 'fileD.txt'];
const data = Promise.all(
  input.map(throat(2, (fileName) => readFile(fileName)))
);

Only 2 files will be read at a time, sometimes limiting parallelism in this way can improve scalability.

Security contact information

To report a security vulnerability, please use the Tidelift security contact. Tidelift will coordinate the fix and disclosure.

License

MIT

throat's People

Contributors

cpojer avatar esrefdurna avatar forbeslindesay avatar greenkeeper[bot] avatar ithinkihaveacat avatar itsjw avatar julien-f avatar lpinca avatar thymikee avatar zkx5xkt avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar

throat's Issues

Handle concurrency of 0

When specifying a concurrency of 0, unexpected results happen. If passing 0, I'd expect an error to be thrown, or the function is not throttled.

Can you add more examples?

Can you add more examples? The featured examples are not actually working examples and I don't know how to implement them.

I suggest something using setTimeout and console.log so you can just copy and paste and have a functioning implementation from the start.

feat: Enqueue without waiting for the result to facilitate backpressure

Is your feature request related to a problem? Please describe.

During async iteration of e.g. a stream, I'd like to be able to signal the stream that the queue is full in order to stop pulling from it.

Describe the solution you'd like

E.g, the following code:

let myThroat = throat(5)
for await (let item of someStream) {
  await myThroat.enqueue(() => process(item))
}

wlll backpressure on pulling from the stream when there are 5 items in the queue already, until one of the items clear

Describe alternatives you've considered

I can't think of an alternative that doesn't somehow have access to the internal queue of throat. It might be possible to do this with standard node streams?

Additional context

N/A

An in-range update of coveralls is breaking the build 🚨

The devDependency coveralls was updated from 3.0.2 to 3.0.3.

🚨 View failing branch.

This version is covered by your current version range and after updating it in your project the build failed.

coveralls is a devDependency of this project. It might not break your production code or affect downstream projects, but probably breaks your build or test tools, which may prevent deploying or publishing.

Status Details
  • continuous-integration/travis-ci/push: The Travis CI build failed (Details).

Release Notes for Dependency security updates

As suggested by NPM and Snyk.

Commits

The new version differs by 1 commits.

  • aa2519c dependency security audit fixes from npm & snyk (#210)

See the full diff

FAQ and help

There is a collection of frequently asked questions. If those don’t help, you can always ask the humans behind Greenkeeper.


Your Greenkeeper Bot 🌴

get rid of promise dependency

As io.js 1.x Promises are native into this runtime, so I hope you would lift the dependency.

throat could be required like this:

var throat = require('throat')(Promise);

// or in node 0.10.x
var throat = require('throat')(require('promise'));

There might be another (with more syntax sugar) versions of this, but this is the simpler that came to mind....

So you can delegate the Promise shim to the user.

edit: mmm... #2

An in-range update of promise is breaking the build 🚨

Version 7.2.0 of promise just got published.

Branch Build failing 🚨
Dependency promise
Current Version 7.1.1
Type devDependency

This version is covered by your current version range and after updating it in your project the build failed.

As promise is “only” a devDependency of this project it might not break production or downstream projects, but “only” your build or test tools – preventing new deploys or publishes.

I recommend you give this issue a high priority. I’m sure you can resolve this 💪

Status Details
  • dependency-ci Dependencies checked Details
  • coverage/coveralls Coverage pending from Coveralls.io Details
  • continuous-integration/travis-ci/push The Travis CI build failed Details

Commits

The new version differs by 6 commits.

See the full diff

Not sure how things should work exactly?

There is a collection of frequently asked questions and of course you may always ask my humans.


Your Greenkeeper Bot 🌴

An in-range update of typescript is breaking the build 🚨

Version 2.5.3 of typescript just got published.

Branch Build failing 🚨
Dependency typescript
Current Version 2.5.2
Type devDependency

This version is covered by your current version range and after updating it in your project the build failed.

As typescript is “only” a devDependency of this project it might not break production or downstream projects, but “only” your build or test tools – preventing new deploys or publishes.

I recommend you give this issue a high priority. I’m sure you can resolve this 💪

Status Details
  • dependency-ci Dependencies checked Details
  • continuous-integration/travis-ci/push The Travis CI build failed Details
  • coverage/coveralls First build on greenkeeper/typescript-2.5.3 at 98.551% Details

Release Notes TypeScript 2.5.3

This release include a set of bug fixes reported against TypeScript 2.5.2. For the complete list of fixed issues, check out the fixed issues query for TypeScript 2.5.3.

Download:

Commits

The new version differs by 51 commits.

  • 94b4f8b Update LKG
  • aa40be6 Merge pull request #18673 from RyanCavanaugh/port18603
  • 6281b05 Report external files in initial case
  • 5538770 Look at correct 'package.json' location for a scoped package (#18580) (#18651)
  • acfd670 Merge pull request #18617 from amcasey/NoModifiers25
  • 9630c46 JavaScript: handle lack of modifiers on extracted method
  • 7b5247f Merge pull request #18605 from RyanCavanaugh/portExternalFilesFixes
  • 6ed9bad Port plugin fixes from master
  • e9a9d2c Merge pull request #18518 from amcasey/ExtractMethodFixes25
  • 27bede8 Stop requiring that the full range of a declaration fall within the
  • f0b7843 Merge pull request #18508 from amcasey/ExtractSingleToken
  • 063e8a7 Merge pull request #18427 from amcasey/GH17869
  • fbb6cd5 Merge pull request #18448 from amcasey/NestedReturn
  • a667b04 Merge pull request #18423 from amcasey/GH18188
  • 334125e Merge pull request #18343 from amcasey/InsertionPosition

There are 51 commits in total.

See the full diff

Not sure how things should work exactly?

There is a collection of frequently asked questions and of course you may always ask my humans.


Your Greenkeeper Bot 🌴

Curious about the implementation decision

Hey Forbes,

Very cool library! I was just curious as to why you used a thunk here: https://github.com/ForbesLindesay/throat/blob/master/index.js#L18 as opposed to a higher order function that executes resolve with the proper arguments.

Why this:

queue.push(new Delayed(resolve, fn, self, args))

instead of

queue.push(function () {
  resolve(run(fn, self, args));
});

And in https://github.com/ForbesLindesay/throat/blob/master/index.js#L26, you could then do:

      var next = queue.shift()
      next()

Not trying to bikeshed on syntax, just genuinely curious if I'm missing something.

Thanks in advance!

Version 10 of node.js has been released

Version 10 of Node.js (code name Dubnium) has been released! 🎊

To see what happens to your code in Node.js 10, Greenkeeper has created a branch with the following changes:

  • Added the new Node.js version to your .travis.yml

If you’re interested in upgrading this repo to Node.js 10, you can open a PR with these changes. Please note that this issue is just intended as a friendly reminder and the PR as a possible starting point for getting your code running on Node.js 10.

More information on this issue

Greenkeeper has checked the engines key in any package.json file, the .nvmrc file, and the .travis.yml file, if present.

  • engines was only updated if it defined a single version, not a range.
  • .nvmrc was updated to Node.js 10
  • .travis.yml was only changed if there was a root-level node_js that didn’t already include Node.js 10, such as node or lts/*. In this case, the new version was appended to the list. We didn’t touch job or matrix configurations because these tend to be quite specific and complex, and it’s difficult to infer what the intentions were.

For many simpler .travis.yml configurations, this PR should suffice as-is, but depending on what you’re doing it may require additional work or may not be applicable at all. We’re also aware that you may have good reasons to not update to Node.js 10, which is why this was sent as an issue and not a pull request. Feel free to delete it without comment, I’m a humble robot and won’t feel rejected 🤖


FAQ and help

There is a collection of frequently asked questions. If those don’t help, you can always ask the humans behind Greenkeeper.


Your Greenkeeper Bot 🌴

An in-range update of typescript is breaking the build 🚨

Version 2.4.1 of typescript just got published.

Branch Build failing 🚨
Dependency typescript
Current Version 2.4.0
Type devDependency

This version is covered by your current version range and after updating it in your project the build failed.

As typescript is “only” a devDependency of this project it might not break production or downstream projects, but “only” your build or test tools – preventing new deploys or publishes.

I recommend you give this issue a high priority. I’m sure you can resolve this 💪

Status Details
  • dependency-ci Dependencies checked Details
  • coverage/coveralls Coverage pending from Coveralls.io Details
  • continuous-integration/travis-ci/push The Travis CI build failed Details

Release Notes TypeScript 2.4

For release notes, check out the release announcement

For new features, check out the What's new in TypeScript 2.4.

For breaking changes, check out the Breaking changes in TypeScript 2.4 page.

For the complete list of fixed issues, check out the fixed issues query for Typescript 2.4.

Download:

Commits

The new version differs by 141 commits.

  • 8b2fe13 Update LKG.
  • 14d95ed Test:Block-scoped definition of Promise works
  • a8846bf Skip block scope check with no error location
  • 44f2336 Merge pull request #16633 from Microsoft/release-2.4_fixIncrementalParsing
  • 4875a27 Add tests
  • 15ef20d Set the structureReused to be safemoudles when dynamic import change
  • 6d33083 Add tests
  • 11b9f6e Wip-fix incremental parsing
  • 2721fd4 In TypeScript code, never bind JSDoc normally, just set parent pointers (#16555) (#16561)
  • 0968ed9 Revert string enum changes (#16569)
  • 096f8cc Update LKG
  • 9241175 Allow running in strict mode (#16557)
  • f49b007 Update LKG
  • f1b0f59 Update version to 2.4.1
  • ed9cde9 Update LKG

There are 141 commits in total.

See the full diff

Not sure how things should work exactly?

There is a collection of frequently asked questions and of course you may always ask my humans.


Your Greenkeeper Bot 🌴

bug: Throat causes a halt in execution under certain conditions

Describe the bug

Throat causes a halt in execution under certain conditions

To Reproduce

const _ = require('lodash');
const throat = require('throat');

async function main() {
  const worker = throat(4, async function (page) {});
  const pagesChunks = _.range(10).map((n) => _.range(100));

  let awaitCount = 0;
  for (const pagesChunk of pagesChunks) {
    console.log(`Await ${awaitCount++}`);
    await Promise.all(pagesChunk.map(worker));
  }

  console.log('Done');
}

main();

This outputs:

Await 0
Await 1
Await 2

Expected behaviour

Expected output:

Await 0
Await 1
Await 2
Await 3
Await 4
Await 5
Await 6
Await 7
Await 8
Await 9
Done

Versions

  • node version: v14.17.5
  • npm/yarn version: 6.14.14
  • package version: throat-6.0.1

Additional context

Messing around with the length of pagesChunk or size can make the script complete normally, or halt earlier or later in the iterations.

[FEATURE REQUEST] flush option

not sure how this would work API-wise, but the idea is to return a promise somewhere that resolves then there is no longer any promises in the queue.

const fn = throat(3, myOtherFunction)

fn(1)
fn(2)
fn(3)
fn(4)

fn.flush().then(doSomethingElse())

it should resolve immediately if there are no more functions in the queue. flush should throw if any of the promises in the queue failed.

An in-range update of testit is breaking the build 🚨

Version 2.1.4 of testit just got published.

Branch Build failing 🚨
Dependency testit
Current Version 2.1.3
Type devDependency

This version is covered by your current version range and after updating it in your project the build failed.

As testit is “only” a devDependency of this project it might not break production or downstream projects, but “only” your build or test tools – preventing new deploys or publishes.

I recommend you give this issue a high priority. I’m sure you can resolve this 💪

Status Details
  • dependency-ci Dependencies checked Details
  • coverage/coveralls Coverage pending from Coveralls.io Details
  • continuous-integration/travis-ci/push The Travis CI build failed Details

Commits

The new version differs by 11 commits.

  • 60e9f16 Update wtfnode ^0.4.0 → ^0.5.4 (#21)
  • 61d71ca fix(package): update ms to version 1.0.0 (#19)
  • 990dba6 fix(package): update wtfnode to version 0.5.0 (#18)
  • 3ffdc22 Merge pull request #12 from ForbesLindesay/greenkeeper-ms-0.7.2
  • 026b361 chore(package): update coveralls to version 2.11.13 (#11)
  • 3c043ab chore(package): update wtfnode to version 0.4.0 (#10)
  • c32c8e6 Merge pull request #14 from ForbesLindesay/greenkeeper-sauce-test-1.1.0
  • aeb2588 Release 2.1.3
  • 9238965 Stop testing on unsupported versions of node.js
  • b4ca44c chore(package): update sauce-test to version 1.1.0
  • c6f4a72 chore(package): update ms to version 0.7.2

See the full diff

Not sure how things should work exactly?

There is a collection of frequently asked questions and of course you may always ask my humans.


Your Greenkeeper Bot 🌴

unhandledRejection is not emitted when function fails and no error handler attached to throat

This leads to errors that are really hard to detect.

You can see that we use throat in pnpm here and we did not add a catch to it. Throat was failing but even the unhandledRejection event was not happening because throat has a default error handler here.

According to NodeJS documentation

The 'rejectionHandled' event is emitted whenever a Promise has been rejected and an error handler was attached to it (using promise.catch(), for example) later than one turn of the Node.js event loop.

The expected behavior would be I think, that unhandledRejection event is fired when no error handler is added for throat outside of the throat source code

Related issue pnpm/pnpm#353

wrap multiple functions in same throat

Hi,

Would you be open to a PR implementing the ability to throttle multiple functions with the same throat instance. Something like

myLimiter = throat(2);
lib.method1 = myLimiter.wrap(lib.method1)
lib.method2 = myLimiter.wrap(lib.method2)

The use case is where e.g. you use one method to read from a db, and another to write to it, but you want to place a limit on the total connections

Suggestion of a new type

Hi,
I work on a typescript project and I use throat module and it's work well, however when I create a lock and I call it with a function which take a argument my IDE say:
"Expected 1 arguments, but got 2."

Example:

const arg = "arg";
const lock = throat(1);
lock(func, arg);

To avoid this problem I added this line in "index.d.ts":

declare function throat(size: number): <TResult>(fn: (args?: any) => Promise<TResult>, args?: any) => Promise<TResult>;

Then I have no this problem anymore if I write like this:

const arg = "arg";
const lock = throat<void>(1);
lock(func, arg);

If you don't mind, can you add something like this?

Best regard

Any way to handle a dynamic input?

Hello, I'm building an upload queue and I'd like to know if it's possible to add more files after calling throat, e.g.:
`

let uploadedFiles = 0
const throat = require('throat')(10)

// uploadQueue initially has 20 files
for (const upload of uploadQueue) {
  throat(async () => {
    // logic for processing upload
    uploadedFiles++;
    console.log({ uploadedFiles, remainingFiles: uploadQueue.length - uploadedFiles })
  })
}

// add another file to the queue
uploadQueue.push(newUpload)

In the above example, I get { uploadedFiles: 20, remainingFiles: 1 } after the initial 20 files are processed and the queue never gets to the new upload. Is there any way to listen to changes in uploadQueue? Or do I have to call throat again every time I add more items to the queue?

Thanks in advance.

An in-range update of promise is breaking the build 🚨

The devDependency promise was updated from 8.0.1 to 8.0.2.

🚨 View failing branch.

This version is covered by your current version range and after updating it in your project the build failed.

promise is a devDependency of this project. It might not break your production code or affect downstream projects, but probably breaks your build or test tools, which may prevent deploying or publishing.

Status Details
  • continuous-integration/travis-ci/push: The Travis CI build failed (Details).

Commits

The new version differs by 7 commits.

  • 1b27a8c Removed unused function (#133)
  • 6a376fe Make the build reproducible (#148)
  • cf30e9f docs(readme): domains link to nodejs
  • 01cf802 Update node versions we test
  • bbf5c44 For better code specifications (#144)
  • 6ead556 Update asap dependency (#143)
  • d141e64 Release 8.0.1

See the full diff

FAQ and help

There is a collection of frequently asked questions. If those don’t help, you can always ask the humans behind Greenkeeper.


Your Greenkeeper Bot 🌴

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.