Comments (22)
They are being returned in the order passed in. The first glob is being excluded because the path does not exist. Should "key" be "hey" in the following... "./fixtures/whatsgoingon/key/isaidhey/whatsgoingon/test.txt"?
from glob-stream.
@azweb76 Probably. Currently they should not be coming back in the order passed in. If they are then something in node-glob is blocking
from glob-stream.
@azweb76, yes, that's an error in the test
from glob-stream.
I corrected the test and I'm having trouble getting things to come out of order. The only explanation I can think of is that combine-stream is ordering the streams it combines but that seems unlikely. We might have to create a more complex dir structure for the "slow" glob in the test
from glob-stream.
Is it ok, when this test fail? I've an issue with it in my current project.
it('should return a correctly ordered file name stream for three globs', function(done) {
var globArray = [
join(__dirname, "./fixtures/**/test.txt"),
join(__dirname, "./fixtures/**/test.coffee"),
join(__dirname, "./fixtures/**/test.js")
];
var stream = gs.create(globArray, {cwd: __dirname});
var files = [];
stream.on('error', done);
stream.on('data', function(file) {
should.exist(file);
should.exist(file.path);
files.push(file);
});
stream.on('end', function() {
files.length.should.equal(3);
path.basename(files[0].path).should.equal('test.txt');
path.basename(files[1].path).should.equal('test.coffee');
path.basename(files[2].path).should.equal('test.js');
done();
});
});
from glob-stream.
@armed Does that fail? If so this is a bug
from glob-stream.
Yes, it fails. At least on my machine. I modified already existing test case: https://github.com/wearefractal/glob-stream/blob/master/test/main.js#L212
from glob-stream.
It started failing when I changed full paths to paths with globstars.
from glob-stream.
@armed Awesome I was having issues making the test fail for ordering. I'll fix them up and start working on this again
from glob-stream.
@contra great news! This fix is a show stopper for my current grunt -> gulp migration.
from glob-stream.
@contra I've been experimenting with ordering of streams. I created drop-in replacement for combine-stream
which preserves ordering. It passes all tests in glob-stream
master plus test which I posted above. Code may be ugly, but I've managed to build all assets in my projects without any issues.
This is what I got (maybe it will help):
var Readable = require('stream').Readable;
var util = require('util');
function OrderedStreams(options) {
options = options || {};
if (Array.isArray(options)) {
options = {streams: options};
}
options.objectMode = true;
Readable.call(this, options);
var streams = options.streams || [];
var self = this;
if (streams.length === 0) {
this.push(null); // no streams, close
} else {
// initial index in list of glob-streams
this._currentIndex = 0;
this._buff = {};
this._openedStreams = streams.length;
streams.forEach(function (s, i) {
s.on('data', function (data) {
// if data emmitted from stream, which is at current index
if (i === self._currentIndex) {
self._currentIndex++;
self.push(data); // push data downstream
} else {
self._buff[i] = data; // or store in buffer for future
}
});
s.on('end', function () {
// if stream ended without any data and it at current index
if (i === self._currentIndex) {
self._currentIndex++; // increment index
}
if (!--self._openedStreams) {
self.push(null); // close OrderedStreams
}
})
s.on('error', function (e) {
self.emit('error', e); // error event downstream
})
});
this.on('finish', function() {
streams.forEach(function (s) {
s.end();
});
});
}
}
util.inherits(OrderedStreams, Readable);
OrderedStreams.prototype._read = function () {
// if we already have stored data push it
var data = this._buff[this._currentIndex];
if (data) {
this._currentIndex++;
this.push(data);
}
}
module.exports = OrderedStreams;
from glob-stream.
@armed got a repo so I can look over the code and the tests?
from glob-stream.
@contra https://github.com/armed/glob-stream, added it in root for convenience.
from glob-stream.
@armed the combined-stream should be in its own package with full test coverage not included in the glob-stream package
from glob-stream.
@contra I wasn't planning to leave it in glob-stream, it's just experiments. If that combine-stream implementation is ok, then I create package.
from glob-stream.
@contra I created package for ordered stream implementation, glob-stream repo with this package is here: https://github.com/armed/glob-stream
from glob-stream.
I ran into something similar and ended up writing stream-series. You can use it like the following:
return series(
gulp.src(['lib/{components,pages}/**/index.js']),
gulp.src(['lib/{components,pages}/**/*.js', '!lib/{components,pages}/**/index.js'])
)
.pipe(concat('build.js'))
.pipe(gulp.dest('public/'))
from glob-stream.
@rschmukler Does the latest glob-stream solve this for you? npm update
should pull it in
from glob-stream.
return gulp.src(['lib/{components,pages}/**/index.js', 'lib/{components,pages}/**/*.js', '!lib/{components,pages}/**/index.js'])
.pipe(concat('build.js'))
.pipe(gulp.dest('public/'))
should work fine now
from glob-stream.
Ah, I am using gulp 3.4.0
which imports vinyl-fs 0.0.1
, which I am guessing does not have the latest glob-stream?
I was still running into the issue but maybe I'm screwing something up...
from glob-stream.
@rschmukler npm update
will update the whole tree. vinyl-fs
hasn't updated but it uses a semver range for glob-stream
so any bug fixes to glob-stream
won't require me to update the whole tree above it
from glob-stream.
Ah woops. Thanks @contra !
from glob-stream.
Related Issues (20)
- Update pumpify HOT 3
- [SPAM] Update glob-parent dependency to 5.1.2 or higher HOT 1
- glob-parent version vulnerability
- Bump glob-parent to resolve ReDos Vulnerability HOT 1
- Dependency vulnerability - Glob-parent HOT 1
- Switch to streamx
- Investigate fast-glob HOT 2
- Lists only 16 or less files when a glob string including no globstar. HOT 15
- BUG: UNC paths aren't resolved correctly HOT 1
- Support custom `unique` filtering through option HOT 1
- Should `create` accept a non-array ignore option HOT 2
- Docs: Note we don't pass `root` option to node-glob
- Remove path normalization? HOT 1
- Implement the streaming globber as a real Readable stream instead of a through stream HOT 2
- Base directory not copying correctly HOT 4
- Flakey tests HOT 1
- How can I use it with gulp? HOT 1
- Can't install glob-stream - ECONNRESET error HOT 1
- How to properly read chunks of all files content into a single stream? HOT 3
- npm audit returns errors. HOT 5
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from glob-stream.