Code Monkey home page Code Monkey logo

queue-promise's Introduction

queue-promise

Default CI/CD Known Vulnerabilities npm package size npm version npm dependency Status npm downloads

queue-promise is a small, dependency-free library for promise-based queues. It will execute enqueued tasks concurrently at a given speed. When a task is being resolved or rejected, an event is emitted.

Installation

$ npm install queue-promise

Usage

With automatic start

import Queue from "queue-promise";

const queue = new Queue({
  concurrent: 1,
  interval: 2000
});

queue.on("start", () => /* … */);
queue.on("stop", () => /* … */);
queue.on("end", () => /* … */);

queue.on("resolve", data => console.log(data));
queue.on("reject", error => console.error(error));

queue.enqueue(asyncTaskA); // resolved/rejected after 0ms
queue.enqueue(asyncTaskB); // resolved/rejected after 2000ms
queue.enqueue(asyncTaskC); // resolved/rejected after 4000ms

Without automatic start

import Queue from "queue-promise";

const queue = new Queue({
  concurrent: 1,
  interval: 2000,
  start: false,
});

queue.enqueue(asyncTaskA);
queue.enqueue(asyncTaskB);
queue.enqueue(asyncTaskC);

while (queue.shouldRun) {
  // 1st iteration after 2000ms
  // 2nd iteration after 4000ms
  // 3rd iteration after 6000ms
  const data = await queue.dequeue();
}

API

new Queue(options)

Create a new Queue instance.

Option Default Description
concurrent 5 How many tasks should be executed in parallel
interval 500 How often should new tasks be executed (in ms)
start true Whether it should automatically execute new tasks as soon as they are added

public .enqueue(tasks)/.add(tasks)

Adds a new task to the queue. A task should be an async function (ES2017) or return a Promise. Throws an error if the provided task is not a valid function.

Example:

async function getRepos(user) {
  return await github.getRepos(user);
}

queue.enqueue(() => {
  return getRepos("userA");
});

queue.enqueue(async () => {
  await getRepos("userB");
});

// …equivalent to:
queue.enqueue([() => getRepos("userA"), () => getRepos("userB")]);

public .dequeue()

Executes n concurrent (based od options.concurrent) promises from the queue. Uses global Promises. Is called automatically if options.start is set to true. Emits resolve and reject events.

Example:

queue.enqueue(() => getRepos("userA"));
queue.enqueue(() => getRepos("userB"));

// If "concurrent" is set to 1, only one promise is executed on dequeue:
const userA = await queue.dequeue();
const userB = await queue.dequeue();

// If "concurrent" is set to 2, two promises are executed concurrently:
const [userA, userB] = await queue.dequeue();

Note:

.dequeue() function throttles (is executed at most once per every options.interval milliseconds).

public .on(event, callback)

Sets a callback for an event. You can set callback for those events: start, stop, resolve, reject, dequeue, end.

Example:

queue.on("dequeue", () => );
queue.on("resolve", data => );
queue.on("reject", error => );
queue.on("start", () => );
queue.on("stop", () => );
queue.on("end", () => );

Note:

dequeue, resolve and reject events are emitted per task. This means that even if concurrent is set to 2, 2 events will be emitted.

public .start()

Starts the queue – it will automatically dequeue tasks periodically. Emits start event.

queue.enqueue(() => getRepos("userA"));
queue.enqueue(() => getRepos("userB"));
queue.enqueue(() => getRepos("userC"));
queue.enqueue(() => getRepos("userD"));
queue.start();

// No need to call `dequeue` – you can just listen for events:
queue.on("resolve", data => );
queue.on("reject", error => );

public .stop()

Forces the queue to stop. New tasks will not be executed automatically even if options.start was set to true. Emits stop event.

public .clear()

Removes all tasks from the queue.

public .state

  • 0: Idle state;
  • 1: Running state;
  • 2: Stopped state;

public .size

Size of the queue.

public .isEmpty

Whether the queue is empty, i.e. there's no tasks.

public .shouldRun

Checks whether the queue is not empty and not stopped.

Tests

$ npm test

Contributing

Development

We have prepared multiple commands to help you develop queue-promise on your own. You will need a local copy of Node.js installed on your machine. Then, install project dependencies using the following command:

$ npm install

Usage

$ npm run <command>

List of commands

Command Description
test Run all test:* commands described below.
test:flow Test Flow types.
test:typescript Test TypeScript types.
test:unit Run unit tests.
test:lint Run linter tests.
defs:flow Build Flow type definitions.
defs:typescript Build TypeScript type definitions.
clean Clean dist directory.
build Build package and generate type definitions.
watch Build package in watch mode.
release Bump package version and generate a CHANGELOG.md file.

License

queue-promise was created and developed by Bartosz Łaniewski. The full list of contributors can be found here. The package is MIT licensed.

Bug reporting

Github Open Issues Github Closed Issues Github Pull Requests

We want contributing to queue-promise to be fun, enjoyable, and educational for anyone, and everyone. Changes and improvements are more than welcome! Feel free to fork and open a pull request. We use Conventional Commits specification for commit messages. If you have found any issues, please report them here - they are being tracked on GitHub Issues.

queue-promise's People

Contributors

bartozzz avatar dependabot-preview[bot] avatar dominiktrenz avatar greenkeeper[bot] avatar pbartela avatar semantic-release-bot avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar

queue-promise's Issues

this.currentlyHandled is smaller than 0

In dequeue launching Promise.all on empty promises causes negative value on this.currentlyHandled
Because of that the bigger the number of requests the more requests is launched making this.options.concurrent irrelevant

How can I achieve the automatic start of a task if space is freed up in the queue?

I expect the queue to refill automatically as it becomes free.

const queue = new Queue({
    concurrent: 3,
    start: true,
    interval: 0
});
queue.on("resolve", n => {
    console.log(n);
});

[10, 10, 1, 1, 1, 1, 1, 1, 1].forEach(n => {
    queue.enqueue(async () => {
        await new Promise((resolve) => {
            setTimeout(resolve, n*1000);
        });
        return n;
    })
})

In this code, I want the fourth task to be executed after the completion of the third one. But in fact, the 4 task will be completed only after the completion of the first two.

return:

1
10, 10
1, 1, 1
1, 1, 1

I expect:

1
1 
1
1
1
1
1
10, 10

How can I achieve the automatic start of a task if space is freed up in the queue?

it can't run in browser

i'm can run it in nodejs

but i cant, run it in breswer

index.js

var QueuePromise = require('queue-promise');
module.exports = QueuePromise;

weback config

const path = require('path');
module.exports = {
    entry: './src/index.js',
    output: {
        path: path.resolve(__dirname, '../dist'),
        publicPath: '/dist/',
        filename: 'queue-promise.js',
        library: 'QueuePromise',
        libraryTarget: 'umd',
        umdNamedDefine: true
    }
};
  <script src="./queue-promise.js"></script>
  <script>
 // var QueuePromise = require('queue-promise')
    var queue = new QueuePromise({
      concurrent: 1, // 并行个数
      interval: 50, // 调用间隔
    });

    queue.on("start", (data) => console.log(data));
    queue.on("stop", (data) => console.log(data));
    queue.on("end", (data) => console.log(data));

    queue.on("resolve", data => console.log(data));
    queue.on("reject", error => console.error(error));

    queue.enqueue(function () {
      return new Promise(res => {
        setTimeout(() => {
          res(11)
        }, 1000);
      })
    });
    queue.enqueue(function () {
      return new Promise(res => {
        setTimeout(() => {
          res(22)
        }, 1000);
      })

    });
    queue.enqueue(function () {
      return new Promise(res => {
        setTimeout(() => {
          res(33)
        }, 1000);
      })

    });
  </script>

Your .dependabot/config.yml contained invalid details

Dependabot encountered the following error when parsing your .dependabot/config.yml:

Automerging is not enabled for this account. You can enable it from the [account settings](https://app.dependabot.com/accounts/Bartozzz/settings) screen in your Dependabot dashboard.

Please update the config file to conform with Dependabot's specification using our docs and online validator.

Internet Explorer support

Right now this library does not work on ie 11 because missing babel plugin" form transform classes".
I can make Pr but I'm not sure if this is something you want to add.

Is .started still a thing?

Documentation shows this can be called to determine Whether the queue is running. However, I see no reference to this variable in the code, nor does it seem to work in the last build (2.2.1).

What am I missing?

Queue randomly stopped working

I literally can't explain it.

export const queue = new Queue({
  concurrent: 20,
  interval: 0,
});

export async function queueTest() {
  const tests: Array<string> = ['1', ... '1000'];

  tests.forEach((testId) => { 
    queue.enqueue(async () => {
      console.log(`Test ${testId}`);
    })
  });
}

I call a function that adds to the queue which lives in a global scope (I think that's how you describe it) in a React app. It worked fine until users told me otherwise. I just can not get it to start. The queued tasks never execute!

I've reproduced my issue here: https://codesandbox.io/s/why-isnt-my-queue-working-4zqke?file=/src/App.js

Queue does not resolve/reject all promises on concurrent execution

For a simple port scanning script I queued executions of port scans expecting each of the scans to either resolve (port opened) or reject (port closed).

Another problem I found is that the queue was not really executing concurrently. After analyzing the reason for this I found the problem on the dequeue() method, which is broken in multiple ways:

return Promise.all(promises) .then(values => { for (let output of values) this.emit("resolve", output); return values;# }) .catch(error => { this.emit("reject", error); return error; }) .then(output => { this.finalize(); return output; });

Promise.all will reject when the first promise rejects! This has two bad effects:

  1. This leads to finalize only being called once even if multiple promises should have been dequeued, leading the currentlyHandled counter to stay one below max, not allowing concurrent execution anymore but just continue with serial execution

  2. No "reject", "resolve" events will be emitted for the executed tasks, that come after the resolve.

I will submit a pull request for a fix.

Event emitter "end" triggers before the queue is finished

I'm trying to queue a bunch of files to upload and when the queue is finished I want to stop it, so I can do other stuff with the data.
On end:

queue.on("end", () => {
    if (queue.isEmpty) {
      console.log("Queue isEmpty", queue.isEmpty);
      queue.stop();
    }
  });

Result:

Starting queue...
 LOG  Queue isEmpty true
 LOG  Queue isEmpty true
 LOG  Queue isEmpty true
 LOG  success: 1 failed: 0

Last log is when the upload is concluded. Maybe I'm thinking this in a wrong way...

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.