Request for ntCard option to specify directory for temporary files about ntcard HOT 5 CLOSED

@mahesh-panchal Hi Mahesh, ntCard does not generate intermediate files. The output histogram files from ntCard are about few hundred bytes.

From your script I see you're using .gz files as inputs. So, the issue may be related to OS TEMP space related to gzip processes. Can you change your TMPDIR to somewhere with enough space such as /var/tmp?

export TMPDIR=/var/tmp

Another solution could be reducing the number of processes in your script, i.e. $SLURM_NPROCS.

from ntcard.

Hmm. Interesting. The problem cannot be the TMPDIR since this is set to use node scratch disk (/scratch/<job_id>) and that has quite a bit of space.

What is the reasoning behind using reducing the number of cores used?

from ntcard.

I've tried some other things too now (including setting TMPDIR), like cd'ing to the output directory, symlinking the files to the output directory, but I'm still not getting output. The cluster hasn't reported an error this time either, but the weird thing is the absence of output now. All the input files definitely exist, and are not broken symlinks.

Is there supposed to be more written to the screen than:

Runtime(sec): 4638.3723

real    77m18.667s
user    262m28.935s
sys     14m13.065s

?

from ntcard.

@mahesh-panchal

What is the reasoning behind using reducing the number of cores used?

Every thread will be working on separate .gz file in parallel. Each thread will fork a gzip process to read the fq.gz files and the higher the number of threads, the more TMP space required for gzip processes in total.

Is there supposed to be more written to the screen than:

I just realized you haven't specified the value(s) of k in your script. Please include it by adding -k option. For example for k=64 use:

time ntcard -k 64 -t $SLURM_NPROCS -p spruce_freq $FQDIR/*.fq.gz

By default the outputs will be written on freq_k$k.hist in the CWD. In your script you have specified spruce_freq as prefix for outputs, so you should see spruce_freq_k64.hist in the current or specified working directory.

from ntcard.

Thank you Hamid,

After including the -k option, it now works. The output is now there.

Thanks again for puzzling this through with me.

Regards,
Mahesh.

from ntcard.