visionmedia / batch Goto Github PK

to produce a single set of progress events, say for example you're resizing images, uploading to s3 etc, you could then combine these batches into one and get a single progress

Why does this code cause the batch to end without doing the jobs?

var Batch = require('batch');

var n = 10;
var batch = new Batch;

batch.concurrency(2);

while (n--) {
    (function(n){
        batch.push(function(done){
            console.log('  start : %s', n);
        arbitrary(done)
        })
    })(n);
}
batch.on('progress', function(e) {
    console.log(e.percent);

});

function arbitrary(done){
    console.log('   done : %s', n);
    done();
}

batch.end();

output is:

  start : 9
   done : -1
10
  start : 8
   done : -1
20
  start : 7
   done : -1
30
  start : 6
   done : -1
40
  start : 5
   done : -1
50
  start : 4
   done : -1
60
  start : 3
   done : -1
70
  start : 2
   done : -1
80
  start : 1
   done : -1
90
  start : 0
   done : -1
100

So the jobs never complete,

thought on adding a Batch#timeout(1000) or something similar?

the try/catch dependency on EventEmitter defeats browserify's module resolution. This is one example of why I kept insisting that we should try to have the dependency resolution in component (for example) look the same as node.

I know you'd like to pretend that browserify doesn't exist, but if the stated goal for the module is "Async batch with concurrency control and progress reporting for nodejs and the browser" then one good metric would be whether it works in browserify.

I'm trying to build a file for AWS lambda.
But because batch is required we get the following error:
Module not found: Error: Can't resolve 'emitter'

This is because emitter isn't a dependency in the package.json file.

Btw isn't this code broken???

try {
  var EventEmitter = require('events').EventEmitter;
  if (!EventEmitter) throw new Error();
} catch (err) {
  var Emitter = require('emitter');
}

The event emitter is exported as default:
const EventEmitter = require('events');

I tried to use this package to iterate over a large array of results from a database call. I need to make an http request for each result. I gave up in the end. I am using Node 4.2.4.

Please provide an example which iterates over a large array of results and processes them in batches.

Your example:

var Batch = require('batch')
  , batch = new Batch;

batch.concurrency(4);

ids.forEach(function(id){
  batch.push(function(done){
    User.get(id, done);
  });
});

batch.on('progress', function(e){

});

batch.end(function(err, users){

});

Please show the creation of the ids array, what User and users are and where they are defined.

Currently I do:

      const batch = new Batch();
      batch.concurrency(20);

I would like to have the ability to do:

      const batch = new Batch({ concurrency: 20 });

it looks like we're checking for a process function here, but it's an object:

$ node -pe 'typeof process'
object

would you take a patch fixing this?

/cc @dougwilson

this shouldn't have been the way it is, we should have used an event object with .total, .pending, .percent etc..

Bundling with esbuild currently fails on this package due to an issue on line 15 of index.js. It references a package called "emitter" that esbuild cannot resolve, thus esbuild fails.

https://github.com/mbostock/queue

• I don't like the method names, but it's neat. something like this might be useful for Batch ?

you didn't even test this at all. you should have left component/batch up at LEAST until you tested this.

In some cases it would be quite useful to make batch collect the errors and not throw up on them. Something like new Batch(dontThrowUp).

var batch = new Batch.throwUp(false);

batch.push(fs.readFile.bind(null, "existing"));
batch.push(fs.readFile.bind(null, "non-existing"));
batch.push(fs.readFile.bind(null, "existing"));

batch.end(function(err, res) {
  console.log(err); // [null, Error, null]
  console.log(res) // [Buffer, null, Buffer]
});

When the callback provided to batch.end throws an error things can go a bit haywire... Here is an example I put together.

const Batch = require('batch')
const batch = new Batch()

batch.push(cb => cb())
batch.push(cb => cb())
batch.push(cb => cb())

var i = 1;

batch.end(function(err) {
  console.log(i++)

  if(err) {
    console.log(err)
  }

  throw "Error"
})

Outputs

1
2
Error

This shows the callback being called twice, once with no error (expected) and once with the error contained in the callback! After doing a bit of digging I found the try/catch block in index.js:117 to be the culprit. I assume this was put in so that synchronous type functions would produce callback style errors. It's a really messy bug to have though because it can catch all downstream errors of the callback.

Heya,

Currently, some of the code for this library looks like:

Batch.prototype.push = function(fn){
  this.fns.push(fn);
  return this;
};

which means I have to do something like....

const requests = [(done) => { done(); }, (done) => { done(); }];
requests.forEach(request => batch.push(request));

but I expect to be able to:

const requests = [(done) => { done(); }, (done) => { done(); }];
batch.fns.push(...requests);

I would like a couple of things:

1. It's not clear if fns is able to be accessed publicly. If it is, then maybe just remove the public method altogether. If not, maybe consider enhancing the method to support spread operations.

2. If things are to stay as they are - consider renaming public method away from push since it doesn't function quite the same as Array.prototype.push. I did not find this intuitive.

I know it's old, but it's nice a simple. New release with a repo at least? ;)