parallel option not recognized about redrun HOT 18 CLOSED

If you want parallel running try using -p before all script names:

redrun -p feed-info stops routes trips stop-times calendar calendar-dates agency

redrun works this way for having ability to mix parallel and series execution in one run.
Is it works for you?

from redrun.

With -p before the scripts, it works, thanks!

There are two reasons I was expecting that -p belongs in the end:

• It seems to be common sense with CLI tools that an option/a flag only applies to the next word. e.g. with foo --bar baz qux, baz would be bar-enabled, but not qux. In my opinion, the current behaviour of redrun is counterintuitive.
• redrun -h tells me that the options belong in the end, and it lists -p as an option.

from redrun.

There seems to be another issue, when run with time npm run build (and the build script calls redrun), It exits immediately while the spawned child processes keep running.

Edit: The issue appears whenever I put anything in front of npm run build, e.g env foo=bar.

from redrun.

There are two reasons I was expecting that -p belongs in the end:

• It seems to be common sense with CLI tools that an option/a flag only applies to the next word. e.g. with foo --bar baz qux, baz would be bar-enabled, but not qux. In my opinion, the current behaviour of redrun is counterintuitive.

Redrun woks im similar way as npm-run-all does
Could you please provide me examples of CLI tools that wait for last argument?

Edit: The issue appears whenever I put anything in front of npm run build, e.g env foo=bar.

It is strange. I use prefixes a lot. What OS, version of node and npm are you using?

from redrun.

Could you please provide me examples of CLI tools that wait for last argument?

well almost every Unix tool, e.g. gcc and ffmpeg.

It is strange. I use prefixes a lot. What OS, version of node and npm are you using?

system_profiler SPSoftwareDataType | grep 'System Version'
# macOS 10.12.5 (16F73)
node -v
# v7.10.0
npm -v
# 4.6.1

from redrun.

I can not reproduce:

coderaiser:~1$ DEBUG=hello redrun build
> node build/feed-info.js > dest/feed_info.txt && node build/stops.js > dest/stops.txt && node build/routes.js > dest/routes.txt && node build/trips.js > dest/trips.txt && node build/stop-times.js > dest/stop_times.txt && node build/calendar.js > dest/calendar.txt && node build/calendar-dates.js > dest/calendar_dates.txt && curl -sL 'https://vbb-gtfs.jannisr.de/latest/agency.txt' > dest/agency.txt
/bin/sh: dest/feed_info.txt: No such file or directory
Command failed: node build/feed-info.js > dest/feed_info.txt && node build/stops.js > dest/stops.txt && node build/routes.js > dest/routes.txt && node build/trips.js > dest/trips.txt && node build/stop-times.js > dest/stop_times.txt && node build/calendar.js > dest/calendar.txt && node build/calendar-dates.js > dest/calendar_dates.txt && curl -sL 'https://vbb-gtfs.jannisr.de/latest/agency.txt' > dest/agency.txt
coderaiser:~1$ npm -v
5.0.1
coderaiser:~$ node -v
v7.10.0
coderaiser:~$ DEBUG=hello npm run build

> @ build /Users/coderaiser
> redrun feed-info stops routes trips stop-times calendar calendar-dates agency -p

> node build/feed-info.js > dest/feed_info.txt && node build/stops.js > dest/stops.txt && node build/routes.js > dest/routes.txt && node build/trips.js > dest/trips.txt && node build/stop-times.js > dest/stop_times.txt && node build/calendar.js > dest/calendar.txt && node build/calendar-dates.js > dest/calendar_dates.txt && curl -sL 'https://vbb-gtfs.jannisr.de/latest/agency.txt' > dest/agency.txt
/bin/sh: dest/feed_info.txt: No such file or directory
Command failed: node build/feed-info.js > dest/feed_info.txt && node build/stops.js > dest/stops.txt && node build/routes.js > dest/routes.txt && node build/trips.js > dest/trips.txt && node build/stop-times.js > dest/stop_times.txt && node build/calendar.js > dest/calendar.txt && node build/calendar-dates.js > dest/calendar_dates.txt && curl -sL 'https://vbb-gtfs.jannisr.de/latest/agency.txt' > dest/agency.txt
npm ERR! code ELIFECYCLE
npm ERR! errno 1
npm ERR! @ build: `redrun feed-info stops routes trips stop-times calendar calendar-dates agency -p`
npm ERR! Exit status 1
npm ERR! 
npm ERR! Failed at the @ build script.
npm ERR! This is probably not a problem with npm. There is likely additional logging output above.
npm WARN Local package.json exists, but node_modules missing, did you mean to install?

npm ERR! A complete log of this run can be found in:
npm ERR!     /Users/coderaiser/.npm/_logs/2017-06-06T11_28_24_248Z-debug.log 

Do you have any errors?

from redrun.

There seems to be another issue, when run with time npm run build (and the build script calls redrun), It exits immediately while the spawned child processes keep running.

Edit: The issue appears whenever I put anything in front of npm run build, e.g env foo=bar.

Do you have any errors?

I'm not talking about the case when a script fails. When I run npm run build, rerdun spawns several child processes but exists for they exit.

from redrun.

I can't reproduce:

coderaiser@cloudcmd:~/redrun$ time redrun build
> babel lib -d legacy/lib && babel bin -d legacy/bin && echo "module.exports = require('./lib/redrun');" > legacy/index.js && echo "module.exports = require('../package');" > legacy/package.js && echo "module.exports = require('../help');" > legacy/help.js
lib/cli-parse.js -> legacy/lib/cli-parse.js
lib/env.js -> legacy/lib/env.js
lib/get-body.js -> legacy/lib/get-body.js
lib/group-parse.js -> legacy/lib/group-parse.js
lib/quote-args.js -> legacy/lib/quote-args.js
lib/redrun.js -> legacy/lib/redrun.js
lib/regexp.js -> legacy/lib/regexp.js
lib/replace.js -> legacy/lib/replace.js
lib/wildcard.js -> legacy/lib/wildcard.js
bin/redrun-completion.js -> legacy/bin/redrun-completion.js
bin/redrun.js -> legacy/bin/redrun.js

real	0m13.687s
user	0m6.372s
sys	0m0.660s

Looks like it is something related to time or sumething.

from redrun.

Looks like it is something related to time or sumething.

I'm pretty sure it's not. Look at this video: https://transfer.sh/tZNbE/redrun.mov

from redrun.

That is really strange. Could you please try this package.json:

{
    "scripts":  {
        "ping1": "ping google.com > /dev/null",
        "ping2": "ping google.com > /dev/null",
        "check": "redrun -p ping1 ping2"
    }
}

With redrun check. Is it work in the same way on your system?

It can be related to calendar-dates.js and other scripts. Could you show me their contents?

from redrun.

Command that you see in output of redrun executed in bin/redrun.js with help of execSync which is works this way:

The child_process.execSync() method is generally identical to child_process.exec() with the exception that the method will not return until the child process has fully closed. When a timeout has been encountered and killSignal is sent, the method won't return until the process has completely exited. Note that if the child process intercepts and handles the SIGTERM signal and doesn't exit, the parent process will wait until the child process has exited.

About your question:

Looking at the parsing logic, I wondered why there handwritten parsing wrapped around yargs-parser…?

All thatyargs-parser does is parsing of command line arguments. After initial parsing redrun should determine what to do with arguments it have: what scripts should it run in parallel, what scripts should it run in series, when it should stop execution if something goes wrong, and when it should proceed. Get arguments is not enough, they should be processed in same way to build resulting command to execute. This is what cli-parse does.

from redrun.

With redrun check. Is it work in the same way on your system?

Works as expected, redrun keeps running forever. (I noticed that it should process.exitCode = 1 if the script couldn't be found).

Content of build/calendar-dates.js:

'use strict'

const pump = require('pump')
const trips = require('vbb-trips')
const through = require('through2')
const csv = require('csv-write-stream')
const {showError} = require('./lib')
const readSchedule = require('./read-schedule')

pump(
	trips.schedules('all'),
	through.obj(readSchedule),
	through.obj(function (sched, _, cb) {
		for (let weekday in sched.exceptions) {
			for (let [time, running] of sched.exceptions[weekday]) {
				this.push({
					service_id: sched.id,
					date: formatDate(time),
					exception_type: running ? '1' : '2'
				})
			}
		}
		cb()
	}),
	csv(),
	process.stdout,
	showError
)

The other files work similarly.

from redrun.

I think it is related to the shell command redrun generates. I expected a bash or sh shell to wait for all background jobs to finish before it exits, but apparently this isn't the case.

I tried with sh -c "node build/feed-info.js > dest/feed_info.txt & node build/stops.js > dest/stops.txt & node build/routes.js > dest/routes.txt & node build/trips.js > dest/trips.txt & node build/stop-times.js > dest/stop_times.txt & node build/calendar.js > dest/calendar.txt & node build/calendar-dates.js > dest/calendar_dates.txt & curl -sL 'https://vbb-gtfs.jannisr.de/latest/agency.txt' > dest/agency.txt" (the long command is the one printed by redrun build) and it exited immediately. So not execSync is to blame here, but the generated command.

from redrun.

Could you please make minimal example to reproduce how redrun immediately exits?

from redrun.

I'm having trouble creating a minimal example. Neither simple long-running node commands nor curl -sL … seems to cause the issue.

Please check the generate-vbb-gtfs repo. It should work/reproduce out of the box.

from redrun.

On ubuntu everything works as expected:

coderaiser@cloudcmd:~/generate-vbb-gtfs$ git clone https://github.com/derhuerst/generate-vbb-gtfs.git
coderaiser@cloudcmd:~/generate-vbb-gtfs$ npm i
coderaiser@cloudcmd:~/generate-vbb-gtfs$ time redrun build
> node build/feed-info.js > dest/feed_info.txt & node build/stops.js > dest/stops.txt & node build/routes.js > dest/routes.txt & node build/trips.js > dest/trips.txt & node build/stop-times.js > dest/stop_times.txt & node build/calendar.js > dest/calendar.txt & node build/calendar-dates.js > dest/calendar_dates.txt & curl -sL 'https://vbb-gtfs.jannisr.de/latest/agency.txt' > dest/agency.txt

real	0m25.037s
user	0m1.600s
sys	0m1.488s
coderaiser@cloudcmd:~/generate-vbb-gtfs$

I can reproduce this on Mac Os. This is very strange: if you set agency in the beginning of a build script, redrun will wait until the end of execution:

Before:

{
    "build": "redrun -p feed-info stops routes trips stop-times calendar calendar-dates agency"
}

After:

{
    "build": "redrun -p agency feed-info stops routes trips stop-times calendar calendar-dates"
}

agency script contains curl request and if the file you downloading will be much bigger (100mb, for example) everything will work good.

It looks like a bug in node.js. As you see, all that redrun does is make command, run it with execSync of node.js and waits for the end of execution. And if node.js says "that is all" - redrun exits.

from redrun.

I think I find a way to fix it, from examples in documentation.
Landed in v5.9.15.

from redrun.

thanks ❤️

from redrun.