gulpjs / glob-stream Goto Github PK

Hello.
Shouldn't this code work?
SO is windows 7 64bit?

  var gs = require('glob-stream')

  var stream = gs.create("./js/*.js")

  stream.on('data', function(file){
    console.log(file) // never reaches here
  });

this one works fine

  glob = require("glob")

  glob("./js/*.js", null, function (er, files) {
    console.log("num files = " + files.length) // print num files = 3
  })

Am I missing something here??

Thanks

While researching globbing alternatives for SocketStream I found the micromatch module. It claims to be a drop-in replacement with a big speedup.

It seems like a good idea to swap minimatch with micromatch.

Providing a glob array correctly builds 2 streams, then tries to combine them with stream-combiner, but the aggregate stream only emits data for the first glob.

This may be a stream-combiner issue: dominictarr/stream-combiner#1

I believe it is the suggested behavior in npm's documentation on scripts property.

This is more of a question: should there be duplicates in the results?

I'm concatenating files and trying to put certain files at the beginning and rest of them to the end. Here index.js will appear twice in the end result:

gulp.src(['app/index.js', 'app/**/*.js']).pipe(concat('all.js'));

I got around it easily by writing unique plugin:

var unique = function() {
    var files = [];
    return es.map(function(file, cb) {
        if (files.indexOf(file.path) === -1) {
            files.push(file.path);
            cb(null, file);
        } else {
            cb();
        }
    });
};

ps. gulp is awesome. I just might be able to skip the whole grunt phase :)

Need to look through the breaking changes, see if they affect our usage and if we need to bump major.

cc @lazd

Using node-glob directly, I can match a file with a call like

require('glob')('testing/*.test', { nocase: true }, (err, files) => console.log(files));

However, using glob-stream, as soon as I pass { nocase: true }, I get zero files in the steam (regardless of whether they match the case of the glob pattern or not). Passing { nocase: false } causes matches to show up again.

require('glob-stream').create('testing/*.test', { nocase: true }).on('data', (file) => console.log(file)).on('end', () => console.log('end'));

The above code only shows end.

I'm running node 4.1.1 on Windows 10, let me know if you need any other debug info.

Found a problem with this. I'll submit a test and fix.

Hi,

lets pretend, I have the following folder structure:

system/
assets/
  |  one/
  |  two/
  |  three/
  |  ...
...

I would now like to stream over all files, but not the assets, but assets/two. In other words: I would like to have all files, except the asset files, that are in the subfolders one/, three/. I could do the following glob to do so:

[
  '**',
  '!assets/one/**',
  '!assets/three/**'
]

But if I would now add another folder within assets I would also need to add it as negative glob within my array. Much more intuitive would be the following glob:

[
  '**',
  '!assets/**',
  'assets/two/**'
]

But that one is not working. I cannot exclude a folder, and include a subfolder back in. But I think that would be very helpful. I am also using node-glob-all, which supports the second glob as expected. It seems like the structure of glob-stream does not support that kind of glob. Is that right? Or am I missing something out? Perhaps you would like to consider using node-glob-all in the future...

Best

PS: I am using gulp, which in the end uses your extension to load the files.

gulpjs/gulp#15

In node-glob, when there are 0 file matches and the "nonull" option is set, the original glob pattern is emitted with the end event. This behaviour is currently lost in glob-stream because the 'end' event handler just closes the stream.

My use case for this functionality was with gulp: I wanted to pass a file name that didn't exist yet to a gulp plugin. Maybe I shouldn't be doing that! I can currently work around the problem in gulp with the following:

gulp.src('./a-file-that-exists', { read : false }) .pipe(rename('the-path-i-want-to-pass-around'))

I can install any other package, but my npm won't install glob-stream (both as dependency for Gulp and manually by npm install --g glob-stream)

It is stuck for a minute, and then, ECONNRESET error appears

npm ERR! code ECONNRESET
npm ERR! errno ECONNRESET
npm ERR! network request to http://registry.npmjs.org/is-extglob/-/is-extglob-2.1.1.tgz failed, reason: read ECONNRESET
npm ERR! network This is a problem related to network connectivity.
npm ERR! network In most cases you are behind a proxy or have bad network settings.
npm ERR! network
npm ERR! network If you are behind a proxy, please make sure that the
npm ERR! network 'proxy' config is set properly.  See: 'npm help config'

npm ERR! A complete log of this run can be found in:
npm ERR!     C:\Users\Chilli\AppData\Roaming\npm-cache\_logs\2017-11-22T17_57_30_004Z-debug.log

NPM log - https://pastebin.com/Ci0AYYzY

I have a directory structure that looks a bit like this:

assets/
  css/
    theme.css
  js/
    app.js
  images/
    logo.png
    content/
      2014-06-29-livestock.jpg
      2014-06-30-building.jpg

All but assets/images/content/* are generated by Gulp. The contents of assets/images/content/ are copied from another location via bash script. I'd like to clean the Gulp-generated files, but leave the files in assets/images/content/.

I've tried the following globs to no avail:

• ["assets", "!assets/images/content"]
• ["assets", "!assets/images/content/**"]
• ["assets/**", "!assets/images/content"]
• ["assets/**", "!assets/images/content/**"]

And eventually found that specifying all the paths explicitly works:

[
  "assets/css",
  "assets/js",
  "assets/images/*",
  "!assets/images/content"
]

We normalize many options passed to node-glob and we are combining ignore and negative globs so it might make sense to normalize a string for ignore to an array (node-glob only accepts an array).

Kind of dependent on #74

Like all the others

I am noticing that the createStream has no tests and relies on the create method to normalize options. Should we remove this method from the external API?

If we are going to remove this method from the external API, should we only export a function instead of exposing a create method?

cc @contra

We support filtering files based on a timestamp in vinyl-fs. Maybe we should push that down to this layer by accepting a filter function in the options and applying it to each match before emitting it on the stream. The specifics about filtering by date could still be in vinyl-fs because it would be generating the filter function.

@contra what do you think about this?

Basically my gulp.src('test/*.js') returns nothing when my process.cwd() contains [] brackets.

Tracing down the gulp dependency tree:

├─┬ [email protected]
│ └─┬ [email protected]
│   ├─┬ [email protected]
│   │ ├─┬ [email protected]

I explicitly installed those versions and here is my test scripts:

require('vinyl-fs')
  .src('test/*.js')
  .on('data', function() { console.log(arguments); }); // no output

require('glob-stream')
  .create('test/*.js')
  .on('data', function() { console.log(arguments); }); // no output

require('glob')('test/*.js', 
  function() { console.log(arguments); }); // lists out my files

Further testing shows that with the current version [email protected] the problem still occurs.

I tried this but it did not work.

gulp.task('css', function() {
    gs(
        [
        'site/bricks/global/global.scss',
        'site/bricks/**/*.scss',
        '!site/bricks/global/domain*.scss',
        'site/bricks/global/domain.my-domain.se.scss'
        ])
        .pipe(concat('style.scss'))
        .pipe(sass().on('error', sass.logError))
        .pipe(gulp.dest('assets/css'))
});

I replaced gulp.src with gs. Why I want to use glob-stream instead of gulp.src is because I want to reinclude my last domain file, which does not work with gulp.src.

I did not figure out how to use it in a gulp task. Maybe someone knows?

The change introduced in 6ef29b6 should get a regression test.

Ref #80 (comment)

While node-glob doesn't implement the ignore option, it would be faster to handle handle globs with the Minimatch class (see sindresorhus/globby#6 (comment) and the comment below).

@contra, I want to follow up on a TODO code comment by you near index.js:85:

    var uniqueStream = unique('path');
    // TODO: set up streaming queue so items come in order
    return aggregate.pipe(uniqueStream);

Are you thinking this is still a possible PR for Gulp now, something that should wait for Gulp 4, or something that doesn't belong in Gulp core.

I see this issue has drifted across several repos and caused more than a little aggravation; I bring it up here because it seems the best point in the code to introduce a consistent way to make gulp.src([...list-of-globs...] align with the ordering documented in gulp's readme:

Files will be concatenated in the order that they are specified in the gulp.src function. For example, to concat ./lib/file3.js, ./lib/file1.js and ./lib/file2.js in that order, the following code will create a task to do that.

I want to get all files in the current folder, including dot files and excluding all files in a certain subfolder.

Example

What I did:

mkdir globtest
cd globtest
npm install glob-stream
touch .gitignore
touch index.js
mkdir subfolder
touch subfolder/subfile.js

Contents of index.js:

require('glob-stream').create(['**', '!node_modules/**'], {dot: true})
  .on('data', function (file) {
    console.log(file.path);
  });

When I run node . dot files in sub folders to node_modules shows up in the output, when I was expecting only index.js, .gitignore, subfolder and subfolder/subfile.js.

UPDATE When changing the patterns to:

require('glob-stream').create(['**', '!node_modules{,/**,/**/.*}'], {dot: true})
  .on('data', function (file) {
    console.log(file.path);
  });

I get the expected result, feels a little bit weird though. Maybe this belongs in the minimatch repo, what do you think?

The tests seem to be a bit flakey. Things like globbing order, etc. These issues can probably be cleaned up.

globby has the same core functionality as glob-stream, except the streaming part. Should these modules be merged, or globby become a dependency of glob-stream?

As far as I can see, globby lacks a streaming API and regex filtering logic. Once those are implemented, both packages should be nearly identical. Then it should be possible to merge their unit tests and that's about it. What do you guys think?

I've also considered keeping globby as non-streamy and glob-stream depend on it, but then globby would still need to implement a streaming or events API to be properly consumed by glob-stream. If globby implements a streaming API, then both packages would be nearly identical (globby with added non-streaming async and sync APIs).

//cc @contra @sindresorhus

This is the only line of code that is not covered yet: https://github.com/wearefractal/glob-stream/blob/1f26389f0408ed33201c17f4658b882242fde881/index.js#L105

The return true doesn't make much sense to me, should we throw an error instead? Then we can just add a test to cover that.

Please upgrade glob-parent to v6.0.1, as they resolved the ReDos Vulnerability, in this version
Reference:

• https://github.com/gulpjs/glob-parent/releases/tag/v6.0.1
• gulpjs/glob-parent#49

I believe it is an issue with this module.

I have two work areas, one older that has 5.3.2 installed, and the new checkout with 5.3.5 installed. One works as expected, and not the other.

In work area with 5.3.5, the issue is that the files are being copied to /tmp/src/xxxxx, and not /tmp/xxxxx

    return gulp.src('src/**/*.js')
        .pipe(eslint())
        .pipe(ngAnnotate())
        .pipe(gulp.dest(conf.path.tmp()));

Even setting the base return gulp.src('src/**/*.js', {base: 'src'}) does not change the output.

Documentation clearly states for gulp that 'src/**/*.js' will not contain the 'src' part of the path, and that it is using glob-stream
https://gulp.readme.io/docs/gulpsrcglobs-options#section-options-base

Trying to use negation in globs can be extremely slow if there are many files in the negated paths (e.g., node_modules). For example:

[ '**/*.js', '!node_modules/**' ]

We're still in the initial phases of a project and while that pattern works, it was causing a lint task to take more than 4s to finish. Changing the pattern to only use positives (but that means manually adding half a dozen paths and files) brought the task time down to 200ms.

spam

this line: https://github.com/wearefractal/glob-stream/blob/6411ea4891d24227f94f302bd52d79584af8fa9b/index.js#L18
Turns all relative paths into absolute paths. When this is passed to node-glob together with a custom root option it tries to resolve them to this custom root. If this root is no the default one, the files can't be found.

Possible solution would be to resolve absolute paths to the root option and then delete that option before passing the options object to node-glob

Rules in Gulp like styles/**/*.{eot,svg,ttf,woff} cause Gulp to crash

Hi,

Before release 5.3.3 and this commit 4a01f00, basePath ended with a path.sep.
And now this behavior has moved to the getBasePath function which lost the path.sep.

On Windows, many gulp plugin not work anymore since this release.

Quoted from here

  if (opt.root && glob[0] === '/') {
    glob = path.resolve(opt.root, '.'+glob);
  } else {
    glob = path.resolve(opt.cwd, glob);
  }

It can be inferred that root takes effect only if glob starts with '/', and relative path are resolved against cwd.
Why treating paths start with and not start with '/' differently?
Is it better to treat them all the same? So that:

root: /a, glob: /b   =>   /a/b 
root: /a, glob: b    =>   /a/b    (not `cwd`/b)

Maybe I am confused but I thought the docs were pretty straight forward. My root project folder has a src folder and a dev folder. In the src folder I have my application files and when I compile the typescript I want the resulting js files generated into the dev folder which is where my web server is loading the built app from.

desired result
myProject
/src
/src/index.html
/src/app/main.ts
gulpfile.js
/dev
/dev/index.html
/dev/app/main.js
/dev/app/main.js.map

I want /src/* to compile into /dev/* but what I am getting is /src folder is compiling into /dev so I have the following..
/dev/src/app/main.js

what I expect is
/dev/app/main.js

gulp.task('compile-ts', function () {
    var tsResult = tsProject.src('src/**/*.ts', {base: 'src'}) 
        .pipe(ts(tsProject)); // transpile the files into .js

    return tsResult.js.pipe(gulp.dest('dev'));
});

tsconfig.json

{
  "compilerOptions": {
    "target": "es5",
    "module": "system",
    "moduleResolution": "node",
    "sourceMap": true,
    "emitDecoratorMetadata": true,
    "experimentalDecorators": true,
    "removeComments": false,
    "noImplicitAny": false,
    "noExternalResolve": false 
  },
  "exclude": [
    "node_modules",
    "typings/main",
    "typings/main.d.ts"
  ]
}

It'd be great if the pumpify dependency got updated, as the currently used version is rather old.

Need to bring the tests inline with other repos, switch from should to expect.

I noticed we explicitly delete this but we state we pass everything through to glob.

It currently does this based on path but it should be customizable. Also, allows for a default function to be supplied and speed up stream creation a tiny bit.

I'd love to not only get chunks with filenames, but also read all files content by chunks (i.e. with fs.createReadStream) and send chunks subsequently into a single common output stream. Wondering what would be the right and shortest way to do so? Thanks.

Now I'm doing it like this:

const outputStream = through.obj();
const filenamesStream = gs(`${this.dirName}/*`);

filenamesStream.on('end', () => { outputStream.end(); })
filenamesStream.pipe(through2.obj(({ path }, _, cb) => {
    const fileStream = fs.createReadStream(path, { encoding: 'utf-8' });
    fileStream.on('end', () => { cb(null); });
    fileStream.pipe(outputStream, { end: false });
}));

return outputStream;

but it's quite ugly.

Since we now have path normalization in vinyl, I think we should drop normalization on the emitted objects to keep it inline with node-glob which doesn't do any normalization. Thoughts?

cc @contra

npm audit returns high errors with dependencies:

full log at https://gist.github.com/GarthDB/73e3f62e9ee4ccde5d22144b4e4ae163

Updating minimatch will fix it, but https://www.npmjs.com/package/glob hasn't been updated. This error impacts everything using glob-stream including vinyl-fs and gulp itself.

I found a high and medium vulnerability for the package glob-parent, the vulnerability references are CVE-2020-28469 (high) and WS-2021-0154 (Medium), SO please requesting you to resolve it as priority. Thanks in advance.

Currently the options are mutated if they are missing defaults.

Webpack can't build 5.3 because it has requires that are not statically evaluable in the micromatch dependency.

This commit introduced micromatch and this problem: 09f1b60

I'm also filing an issue with micromatch 👍

npm WARN deprecated [email protected]: Please update to minimatch 3.0.2 or higher to avoid a RegExp DoS issue

Which causes this:

[19:59:48] Starting 'zip'...
[19:59:49] 'zip' errored after 1.51 s
[19:59:49] RangeError: Maximum call stack size exceeded
    at String.replace (native)
    at globUnescape (/$REPLACED$/node_modules/gulp/node_modules/vinyl-fs/node_modules/glob-stream/node_modules/glob/node_modules/minimatch/minimatch.js:919:12)
    at Minimatch.parse (/$REPLACED$/node_modules/gulp/node_modules/vinyl-fs/node_modules/glob-stream/node_modules/glob/node_modules/minimatch/minimatch.js:618:12)
    at Array.map (native)
    at Minimatch.<anonymous> (/$REPLACED$/node_modules/gulp/node_modules/vinyl-fs/node_modules/glob-stream/node_modules/glob/node_modules/minimatch/minimatch.js:174:14)
    at Array.map (native)
    at Minimatch.make (/$REPLACED$/node_modules/gulp/node_modules/vinyl-fs/node_modules/glob-stream/node_modules/glob/node_modules/minimatch/minimatch.js:173:13)
    at new Minimatch (/$REPLACED$/node_modules/gulp/node_modules/vinyl-fs/node_modules/glob-stream/node_modules/glob/node_modules/minimatch/minimatch.js:128:8)
    at setopts (/$REPLACED$/node_modules/gulp/node_modules/vinyl-fs/node_modules/glob-stream/node_modules/glob/common.js:112:20)
    at new Glob (/$REPLACED$/node_modules/gulp/node_modules/vinyl-fs/node_modules/glob-stream/node_modules/glob/glob.js:117:3)

Without a data event on stream object (as showed in your README.md), can we use it in a sync way?