piped commands that fail don't trigger an error in subprocess.check_call about bcbio-nextgen HOT 3 CLOSED

Brent;
Nice one. Thanks for the pipefail tip. The piping problem was annoying me but I didn't know any tricks to work around it. What I'll plan to do is provide a general function to call out to subprocess that:

• Tries to find an installed bash and use it if present.
• Sets executable when calling subprocess (if bash is found).
• Adds the pipefail command (if bash is found).
• Logs commands for full provenance of results (work in progress here: https://github.com/chapmanb/bcbio-nextgen/blob/master/bcbio/provenance/diagnostics.py#L12)

I'll update this once I have a chance to work this through. Thanks again.

from bcbio-nextgen.

maybe this could be implemented by some check_?? decoroator.

e.g. for this, it'd be check_bam_ok for other functions, it could be check_file_exists or check_file_non_empty

they could all expect a path to a file as the only argument and raise an Exception if the appropriate checks did not pass.

from bcbio-nextgen.

Brent;
Thanks for all the great ideas. I checked in improvements which hopefully handle this, using your check function approach. I agree this is a lot less magic, which is good. My only worry is if failed commands result in partially truncated intermediates, which will be harder to diagnose but this is a good starting point. Let me know if you run into any issues with this and feel free to re-open.

from bcbio-nextgen.