gulpjs / glob-stream Goto Github PK
View Code? Open in Web Editor NEWReadable streamx interface over anymatch.
License: MIT License
Readable streamx interface over anymatch.
License: MIT License
In node-glob, when there are 0 file matches and the "nonull" option is set, the original glob pattern is emitted with the end event. This behaviour is currently lost in glob-stream because the 'end' event handler just closes the stream.
My use case for this functionality was with gulp: I wanted to pass a file name that didn't exist yet to a gulp plugin. Maybe I shouldn't be doing that! I can currently work around the problem in gulp with the following:
gulp.src('./a-file-that-exists', { read : false }) .pipe(rename('the-path-i-want-to-pass-around'))
We support filtering files based on a timestamp in vinyl-fs. Maybe we should push that down to this layer by accepting a filter function in the options and applying it to each match before emitting it on the stream. The specifics about filtering by date could still be in vinyl-fs because it would be generating the filter function.
@contra what do you think about this?
Hello.
Shouldn't this code work?
SO is windows 7 64bit?
var gs = require('glob-stream')
var stream = gs.create("./js/*.js")
stream.on('data', function(file){
console.log(file) // never reaches here
});
this one works fine
glob = require("glob")
glob("./js/*.js", null, function (er, files) {
console.log("num files = " + files.length) // print num files = 3
})
Am I missing something here??
Thanks
Please upgrade glob-parent to v6.0.1, as they resolved the ReDos Vulnerability, in this version
Reference:
@contra, I want to follow up on a TODO code comment by you near index.js:85
:
var uniqueStream = unique('path');
// TODO: set up streaming queue so items come in order
return aggregate.pipe(uniqueStream);
Are you thinking this is still a possible PR for Gulp now, something that should wait for Gulp 4, or something that doesn't belong in Gulp core.
I see this issue has drifted across several repos and caused more than a little aggravation; I bring it up here because it seems the best point in the code to introduce a consistent way to make gulp.src([...list-of-globs...] align with the ordering documented in gulp's readme:
Files will be concatenated in the order that they are specified in the gulp.src function. For example, to concat ./lib/file3.js, ./lib/file1.js and ./lib/file2.js in that order, the following code will create a task to do that.
Trying to use negation in globs can be extremely slow if there are many files in the negated paths (e.g., node_modules
). For example:
[ '**/*.js', '!node_modules/**' ]
We're still in the initial phases of a project and while that pattern works, it was causing a lint task to take more than 4s to finish. Changing the pattern to only use positives (but that means manually adding half a dozen paths and files) brought the task time down to 200ms.
Without a data
event on stream object (as showed in your README.md), can we use it in a sync way?
Quoted from here
if (opt.root && glob[0] === '/') {
glob = path.resolve(opt.root, '.'+glob);
} else {
glob = path.resolve(opt.cwd, glob);
}
It can be inferred that root
takes effect only if glob starts with '/', and relative path are resolved against cwd
.
Why treating paths start with and not start with '/' differently?
Is it better to treat them all the same? So that:
root: /a, glob: /b => /a/b
root: /a, glob: b => /a/b (not `cwd`/b)
spam
This is more of a question: should there be duplicates in the results?
I'm concatenating files and trying to put certain files at the beginning and rest of them to the end. Here index.js
will appear twice in the end result:
gulp.src(['app/index.js', 'app/**/*.js']).pipe(concat('all.js'));
I got around it easily by writing unique
plugin:
var unique = function() {
var files = [];
return es.map(function(file, cb) {
if (files.indexOf(file.path) === -1) {
files.push(file.path);
cb(null, file);
} else {
cb();
}
});
};
ps. gulp is awesome. I just might be able to skip the whole grunt phase :)
Hi,
lets pretend, I have the following folder structure:
system/
assets/
| one/
| two/
| three/
| ...
...
I would now like to stream over all files, but not the assets, but assets/two
. In other words: I would like to have all files, except the asset files, that are in the subfolders one/
, three/
. I could do the following glob to do so:
[
'**',
'!assets/one/**',
'!assets/three/**'
]
But if I would now add another folder within assets
I would also need to add it as negative glob within my array. Much more intuitive would be the following glob:
[
'**',
'!assets/**',
'assets/two/**'
]
But that one is not working. I cannot exclude a folder, and include a subfolder back in. But I think that would be very helpful. I am also using node-glob-all, which supports the second glob as expected. It seems like the structure of glob-stream does not support that kind of glob. Is that right? Or am I missing something out? Perhaps you would like to consider using node-glob-all in the future...
Best
PS: I am using gulp, which in the end uses your extension to load the files.
globby has the same core functionality as glob-stream, except the streaming part. Should these modules be merged, or globby become a dependency of glob-stream?
As far as I can see, globby lacks a streaming API and regex filtering logic. Once those are implemented, both packages should be nearly identical. Then it should be possible to merge their unit tests and that's about it. What do you guys think?
I've also considered keeping globby as non-streamy and glob-stream depend on it, but then globby would still need to implement a streaming or events API to be properly consumed by glob-stream. If globby implements a streaming API, then both packages would be nearly identical (globby with added non-streaming async and sync APIs).
//cc @contra @sindresorhus
While node-glob doesn't implement the ignore
option, it would be faster to handle handle globs with the Minimatch class (see sindresorhus/globby#6 (comment) and the comment below).
Hi,
Before release 5.3.3 and this commit 4a01f00, basePath ended with a path.sep
.
And now this behavior has moved to the getBasePath
function which lost the path.sep
.
On Windows, many gulp plugin not work anymore since this release.
Since we now have path normalization in vinyl
, I think we should drop normalization on the emitted objects to keep it inline with node-glob
which doesn't do any normalization. Thoughts?
cc @contra
While researching globbing alternatives for SocketStream I found the micromatch module. It claims to be a drop-in replacement with a big speedup.
It seems like a good idea to swap minimatch with micromatch.
Maybe I am confused but I thought the docs were pretty straight forward. My root project folder has a src folder and a dev folder. In the src folder I have my application files and when I compile the typescript I want the resulting js files generated into the dev folder which is where my web server is loading the built app from.
desired result
myProject
/src
/src/index.html
/src/app/main.ts
gulpfile.js
/dev
/dev/index.html
/dev/app/main.js
/dev/app/main.js.map
I want /src/* to compile into /dev/* but what I am getting is /src folder is compiling into /dev so I have the following..
/dev/src/app/main.js
what I expect is
/dev/app/main.js
gulp.task('compile-ts', function () {
var tsResult = tsProject.src('src/**/*.ts', {base: 'src'})
.pipe(ts(tsProject)); // transpile the files into .js
return tsResult.js.pipe(gulp.dest('dev'));
});
tsconfig.json
{
"compilerOptions": {
"target": "es5",
"module": "system",
"moduleResolution": "node",
"sourceMap": true,
"emitDecoratorMetadata": true,
"experimentalDecorators": true,
"removeComments": false,
"noImplicitAny": false,
"noExternalResolve": false
},
"exclude": [
"node_modules",
"typings/main",
"typings/main.d.ts"
]
}
I'd love to not only get chunks with filenames, but also read all files content by chunks (i.e. with fs.createReadStream
) and send chunks subsequently into a single common output stream. Wondering what would be the right and shortest way to do so? Thanks.
Now I'm doing it like this:
const outputStream = through.obj();
const filenamesStream = gs(`${this.dirName}/*`);
filenamesStream.on('end', () => { outputStream.end(); })
filenamesStream.pipe(through2.obj(({ path }, _, cb) => {
const fileStream = fs.createReadStream(path, { encoding: 'utf-8' });
fileStream.on('end', () => { cb(null); });
fileStream.pipe(outputStream, { end: false });
}));
return outputStream;
but it's quite ugly.
I want to get all files in the current folder, including dot files and excluding all files in a certain subfolder.
Example
What I did:
mkdir globtest
cd globtest
npm install glob-stream
touch .gitignore
touch index.js
mkdir subfolder
touch subfolder/subfile.js
Contents of index.js
:
require('glob-stream').create(['**', '!node_modules/**'], {dot: true})
.on('data', function (file) {
console.log(file.path);
});
When I run node .
dot files in sub folders to node_modules
shows up in the output, when I was expecting only index.js, .gitignore, subfolder and subfolder/subfile.js
.
UPDATE When changing the patterns to:
require('glob-stream').create(['**', '!node_modules{,/**,/**/.*}'], {dot: true})
.on('data', function (file) {
console.log(file.path);
});
I get the expected result, feels a little bit weird though. Maybe this belongs in the minimatch
repo, what do you think?
I tried this but it did not work.
gulp.task('css', function() {
gs(
[
'site/bricks/global/global.scss',
'site/bricks/**/*.scss',
'!site/bricks/global/domain*.scss',
'site/bricks/global/domain.my-domain.se.scss'
])
.pipe(concat('style.scss'))
.pipe(sass().on('error', sass.logError))
.pipe(gulp.dest('assets/css'))
});
I replaced gulp.src
with gs
. Why I want to use glob-stream
instead of gulp.src
is because I want to reinclude my last domain file, which does not work with gulp.src
.
I did not figure out how to use it in a gulp task. Maybe someone knows?
Need to look through the breaking changes, see if they affect our usage and if we need to bump major.
It'd be great if the pumpify dependency got updated, as the currently used version is rather old.
We normalize many options passed to node-glob
and we are combining ignore
and negative globs so it might make sense to normalize a string for ignore
to an array (node-glob
only accepts an array).
Kind of dependent on #74
Basically my gulp.src('test/*.js')
returns nothing when my process.cwd()
contains []
brackets.
Tracing down the gulp
dependency tree:
├─┬ [email protected]
│ └─┬ [email protected]
│ ├─┬ [email protected]
│ │ ├─┬ [email protected]
I explicitly installed those versions and here is my test scripts:
require('vinyl-fs')
.src('test/*.js')
.on('data', function() { console.log(arguments); }); // no output
require('glob-stream')
.create('test/*.js')
.on('data', function() { console.log(arguments); }); // no output
require('glob')('test/*.js',
function() { console.log(arguments); }); // lists out my files
Further testing shows that with the current version [email protected]
the problem still occurs.
Like all the others
Providing a glob array correctly builds 2 streams, then tries to combine them with stream-combiner, but the aggregate stream only emits data for the first glob.
This may be a stream-combiner issue: dominictarr/stream-combiner#1
Currently the options are mutated if they are missing defaults.
Found a problem with this. I'll submit a test and fix.
This is the only line of code that is not covered yet: https://github.com/wearefractal/glob-stream/blob/1f26389f0408ed33201c17f4658b882242fde881/index.js#L105
The return true
doesn't make much sense to me, should we throw an error instead? Then we can just add a test to cover that.
Using node-glob directly, I can match a file with a call like
require('glob')('testing/*.test', { nocase: true }, (err, files) => console.log(files));
However, using glob-stream, as soon as I pass { nocase: true }
, I get zero files in the steam (regardless of whether they match the case of the glob pattern or not). Passing { nocase: false }
causes matches to show up again.
require('glob-stream').create('testing/*.test', { nocase: true }).on('data', (file) => console.log(file)).on('end', () => console.log('end'));
The above code only shows end
.
I'm running node 4.1.1 on Windows 10, let me know if you need any other debug info.
I noticed we explicitly delete this but we state we pass everything through to glob.
Need to bring the tests inline with other repos, switch from should
to expect
.
I am noticing that the createStream
has no tests and relies on the create
method to normalize options. Should we remove this method from the external API?
If we are going to remove this method from the external API, should we only export a function instead of exposing a create
method?
cc @contra
The change introduced in 6ef29b6 should get a regression test.
Webpack can't build 5.3 because it has requires that are not statically evaluable in the micromatch dependency.
This commit introduced micromatch and this problem: 09f1b60
I'm also filing an issue with micromatch 👍
I have a directory structure that looks a bit like this:
assets/
css/
theme.css
js/
app.js
images/
logo.png
content/
2014-06-29-livestock.jpg
2014-06-30-building.jpg
All but assets/images/content/*
are generated by Gulp. The contents of assets/images/content/
are copied from another location via bash script. I'd like to clean the Gulp-generated files, but leave the files in assets/images/content/
.
I've tried the following globs to no avail:
["assets", "!assets/images/content"]
["assets", "!assets/images/content/**"]
["assets/**", "!assets/images/content"]
["assets/**", "!assets/images/content/**"]
And eventually found that specifying all the paths explicitly works:
[
"assets/css",
"assets/js",
"assets/images/*",
"!assets/images/content"
]
npm audit
returns high errors with dependencies:
full log at https://gist.github.com/GarthDB/73e3f62e9ee4ccde5d22144b4e4ae163
Updating minimatch will fix it, but https://www.npmjs.com/package/glob hasn't been updated. This error impacts everything using glob-stream including vinyl-fs and gulp itself.
npm WARN deprecated [email protected]: Please update to minimatch 3.0.2 or higher to avoid a RegExp DoS issue
Which causes this:
[19:59:48] Starting 'zip'...
[19:59:49] 'zip' errored after 1.51 s
[19:59:49] RangeError: Maximum call stack size exceeded
at String.replace (native)
at globUnescape (/$REPLACED$/node_modules/gulp/node_modules/vinyl-fs/node_modules/glob-stream/node_modules/glob/node_modules/minimatch/minimatch.js:919:12)
at Minimatch.parse (/$REPLACED$/node_modules/gulp/node_modules/vinyl-fs/node_modules/glob-stream/node_modules/glob/node_modules/minimatch/minimatch.js:618:12)
at Array.map (native)
at Minimatch.<anonymous> (/$REPLACED$/node_modules/gulp/node_modules/vinyl-fs/node_modules/glob-stream/node_modules/glob/node_modules/minimatch/minimatch.js:174:14)
at Array.map (native)
at Minimatch.make (/$REPLACED$/node_modules/gulp/node_modules/vinyl-fs/node_modules/glob-stream/node_modules/glob/node_modules/minimatch/minimatch.js:173:13)
at new Minimatch (/$REPLACED$/node_modules/gulp/node_modules/vinyl-fs/node_modules/glob-stream/node_modules/glob/node_modules/minimatch/minimatch.js:128:8)
at setopts (/$REPLACED$/node_modules/gulp/node_modules/vinyl-fs/node_modules/glob-stream/node_modules/glob/common.js:112:20)
at new Glob (/$REPLACED$/node_modules/gulp/node_modules/vinyl-fs/node_modules/glob-stream/node_modules/glob/glob.js:117:3)
I found a high and medium vulnerability for the package glob-parent, the vulnerability references are CVE-2020-28469 (high) and WS-2021-0154 (Medium), SO please requesting you to resolve it as priority. Thanks in advance.
I believe it is the suggested behavior in npm's documentation on scripts
property.
this line: https://github.com/wearefractal/glob-stream/blob/6411ea4891d24227f94f302bd52d79584af8fa9b/index.js#L18
Turns all relative paths into absolute paths. When this is passed to node-glob
together with a custom root
option it tries to resolve them to this custom root. If this root
is no the default one, the files can't be found.
Possible solution would be to resolve absolute paths to the root
option and then delete that option before passing the options object to node-glob
The tests seem to be a bit flakey. Things like globbing order, etc. These issues can probably be cleaned up.
I can install any other package, but my npm won't install glob-stream (both as dependency for Gulp and manually by npm install --g glob-stream
)
It is stuck for a minute, and then, ECONNRESET error appears
npm ERR! code ECONNRESET
npm ERR! errno ECONNRESET
npm ERR! network request to http://registry.npmjs.org/is-extglob/-/is-extglob-2.1.1.tgz failed, reason: read ECONNRESET
npm ERR! network This is a problem related to network connectivity.
npm ERR! network In most cases you are behind a proxy or have bad network settings.
npm ERR! network
npm ERR! network If you are behind a proxy, please make sure that the
npm ERR! network 'proxy' config is set properly. See: 'npm help config'
npm ERR! A complete log of this run can be found in:
npm ERR! C:\Users\Chilli\AppData\Roaming\npm-cache\_logs\2017-11-22T17_57_30_004Z-debug.log
NPM log - https://pastebin.com/Ci0AYYzY
cc @lazd
It currently does this based on path
but it should be customizable. Also, allows for a default function to be supplied and speed up stream creation a tiny bit.
Ref #80 (comment)
Rules in Gulp like styles/**/*.{eot,svg,ttf,woff}
cause Gulp to crash
I believe it is an issue with this module.
I have two work areas, one older that has 5.3.2 installed, and the new checkout with 5.3.5 installed. One works as expected, and not the other.
In work area with 5.3.5, the issue is that the files are being copied to /tmp/src/xxxxx, and not /tmp/xxxxx
return gulp.src('src/**/*.js')
.pipe(eslint())
.pipe(ngAnnotate())
.pipe(gulp.dest(conf.path.tmp()));
Even setting the base return gulp.src('src/**/*.js', {base: 'src'}) does not change the output.
Documentation clearly states for gulp that 'src/**/*.js' will not contain the 'src' part of the path, and that it is using glob-stream
https://gulp.readme.io/docs/gulpsrcglobs-options#section-options-base
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.