`gulp.src(..).pipe(gulp.watch(..))` about gulp HOT 23 CLOSED

Or maybe just make gulp.watch() inline, and emit new files whenever it discovers them or has them changed? Then it could replace gulp.src().

gulp.watch("./client/js/*.coffee", {ignore: ["vendor"]}) // just like gulp.src, except watches files
    .pipe(coffee())
    .pipe(minify())
    .pipe(gulp.dest("./public/js"));

from gulp.

@gratimax - Love it.

from gulp.

@gratimax I do have concerns though that incremental builds will complicate plugin development. I need to think about this for a bit.

from gulp.

@contra Not sure what you mean by incremental builds.

EDIT: nevermind

from gulp.

@gratimax - The expected behavior of this would be that when a file gets modified it gets emitted down the stream again. The problem arises when you have something that works against sets of files in your task (like gulp-concat). This would also totally screw up task dependencies since a task with .watch() in it really never ends so if something depends on it then it will never be executed.

from gulp.

@contra One possible solution is to just use new the newly defined trigger(from issue #32, I was thinking of that all along) whenever a watch is used, so like your example in that issue:

gulp.task 'default', ->
  gulp.watch("./client/js/**")
    .pipe(gulp.trigger 'js')
    .pipe(gulp.trigger 'coffee')

This does require having to change how tasks work, so that they'd function more like event stream maps than anything.

Another possible solution is adding some stuff to gulp-util that allows you to make 'file blobs', or collections of files that check which files are recomputed, and always send the full file list down the pipe. You could create a file blob from gulp-util that would read the incoming stream and automatically add new files. Any change in the files list, and the whole file list is sent down again. This would also make passing things along easier.

Example for gulp-concat:

var es = require('event-stream'),
  path = require('path'),
  util = require('gulp-util');

module.exports = function(fileName){
  if (!fileName) throw new Error("Missing fileName option for gulp-concat");

  var blob = new util.FileBlob(); // Create a file blob

  function concatFiles(files, cb){ // files is an array that blob stores

    var joinedContents = files.map(function(file){
      return file.contents;
    }).join('');

    var joinedPath = path.join(path.dirname(files[0].path), fileName);

    var joinedFile = {
      shortened: fileName,
      path: joinedPath,
      contents: joinedContents
    };

    cb(null, joinedFile);
  }

  return es.pipeline(
    blob.add, // blob.add adds a file to the blob
    es.map(concatFiles) // our concat function
  ); 
};

The resulting code is shorter and arguably more concise. Anyway, this is a pretty big change. I think I may have to pull this into its own issue.

from gulp.

The use-case that created this is I want to gulp.src('/path/to/some.thing') then pipe that through a filter before watching. E.g. perhaps I want to watch **/*.js except **/external/**. Right now minimatch doesn't support globs like {**/*.js,!**/external/**} because it presumes ! will be the first character in the whole thing, not the first character in each match expression. It also doesn't support the ,{ignore:['**/external/**']} syntax anymore. Thus, there's no good way to say "watch part of this path without watching all of it".

from gulp.

@robrich I'm working on standardizing the arguments passed to .watch and .src to support arrays of globs and arrays of ignores

from gulp.

Given that you'd usually apply the same transforms to the same input files whether you compile them once (.src) or you watch them while in dev (.watch), have you considered making that an option of the execution rather than a part of the pipeline, for instance as a CLI option (gulp --watch <taskname>)?

from gulp.

@theefer I'm still thinking about the best way to do that. It is annoying to have to keep pasting the same paths around

from gulp.

Why not create glob streams ? A stream of gulp.src objects. Plugins are functions we run to return a stream to pipe. So, we could do that:

gulp.watch("./client/js/*.coffee", {ignore: ["vendor"]}) // set a stream of gulp.src results
    .pipe(coffee) // just provide the function
    .pipe(gulp.wrap(header, "This is a header")) // helper to run a function with the given parameters
    .pipe(minify)
    .pipe(gulp.wrap(gulp.dest,"./public/js"));

That way, we could end streams, concat files etc... without annoying side effects.

from gulp.

Okay so I think we can do this but it will only be for plugins that work with incremental builds. Things that need every file at once (concat, bundling, etc.) will need to use the traditional .src() stuff.

from gulp.

This will be added into vinyl-fs and brought into gulp when that is.

from gulp.

Based on @gratimax 's first proposal, ideally something like the following would be awesome:

gulp.task('default', function() {

    var target = './**/*.coffee';

    // hypothetical "stream maker"
    // to avoid repetition
    var gulpCoffee = gulp.factory() 
                         .pipe(coffee())
                         .pipe(minify())
                         .pipe(gulp.dest("./public/js"));

    // Process all coffee files.
    gulp.src(target)
        .gulpCoffee();

    gulp.watch("./client/js/*.coffee", {ignore: ["vendor"]}) // just like gulp.src, except watches files
        .gulpCoffee();

});

from gulp.

@dashed, couldn't you just do that right now (prettiest in coffeescript):

coffeePipe = (stream) ->
  stream.pipe(coffee())
    .pipe(minify())
    .pipe(gulp.dest("./public/js"))
}
gulp.task('default', ->
  coffeePipe gulp.src('**/*.coffee')
  coffeePipe gulp.watch('client/js/*.coffee', {ignore: ['vendor'])
}

I guess this solution isn't the best though for such a common use case.

What about gulp.watchAndGather to solve this problem of watching one file then wanting all of the glob?

It would trigger an src when any of the files in the glob change? There's probably a prettier way...

gulp.watchAndGather("./client/js/*.coffee", {ignore: ["vendor"]}) // just like gulp.src, except watches files
    .pipe(coffee())
    .pipe(minify())
    .pipe(gulp.dest("./public/js"));

from gulp.

@ajoslin I already have something; a little similar. It's a little hacky solution (but not too hacky for it to be a problem): https://github.com/gobanjs/gojs/blob/rewrite/gulpfile.js

Your coffeescript proposal is interesting.

Another alternative use case is:

   //... 

    // hypothetical "stream maker"
    // to avoid repetition
    var gulpCoffee = gulp.factory() 
                         .pipe(coffee())
                         .pipe(minify())
                         .pipe(gulp.dest("./public/js"));

    // Process all coffee files.
    gulp.src(target)
        .pipe(gulpCoffee());

    gulp.watch("./client/js/*.coffee", {ignore: ["vendor"]}) // just like gulp.src, except watches files
        .pipe(gulpCoffee());

   // ...

The difference is that gulpCoffee can be pipe'd. I think this may be possible.

from gulp.

Just realized eventStream.pipeline() is your gulp.factory() idea :-)

This here works great:

function scripts() {
  return es.pipeline(
    coffee({bare: true}),
    uglify(),
    concat('script.js'),
    gulp.dest('dist/')
  );
}
gulp.task('default', function() {
  gulp.src('*.coffee').pipe(scripts());
});

I think we've gotten a little off-topic though .. I think if we can just get watch to be pipable we could do this really well.

from gulp.

We can continue this on another issue if we're off topic. I'm not an expert on streaming; since I adopted gulp only a few days ago. I'm still wrapping my head around streaming.

If the idiom is that simple, I don't see a reason for it not to be part of gulp once gulp.watch(...) is more fleshed out.

from gulp.

@dashed @ajoslin

If enough people want it I could probably throw this in gulp-util and/or as its own module

gulp.combine = function(){
  var args = arguments;
  return function(){
    return es.pipeline.apply(es, args);
  };
};

Usage:

var scripts = gulp.combine(
    coffee({bare: true}),
    uglify(),
    concat('script.js'),
    gulp.dest('dist/')
);

gulp.task('default', function() {
  gulp.src('*.coffee').pipe(scripts());
});

from gulp.

That would be useful to keep things DRY.

from gulp.

@dashed Moving this to gulpjs/gulp-util#16

from gulp.

Moving this to #84

from gulp.

Also somebody should put that combine stuff in the recipes section of the wiki

from gulp.