simexp / cbrain-plugins-psom Goto Github PK

Should be done by @ttaa9 if possible.

• BIDS support in PSOM
• BIDS support in CBRAIN: view (list files, print validation errors).

CBRAIN has a framework to display QC panels specific to a particular file type, and to tag these files accordingly. A QC framework can be implemented by:

• creating a specific type of PSOM or Niak results. Done. All files ending with ".niak" will be associated with a particular CBRAIN file type that will have the Niak viewer and QC panel.
• The viewer and QC panels associated to this file type will show HTML pages produced by Niak. These webpages will be generated by Niak once SIMEXP/niak#176 is implemented.

@pbellec also suggested that:

• QC tags are saved in the data. At the moment this cannot be done easily but a CSV file containing all the QC tags can be exported from the file list.
• Comments can be added to QC tags. Available by default in CBRAIN.
• Viewer is also used while pipeline is running, to monitor progression.

Somehow the file name is replaced by a serialized Ruby object:

 /usr/local/bin/niak_cmd.py Niak_fmri_preprocess --file_in #<BidsDataset:0x00000005bd1258> --folder_out  --opt-psom-max_queued 500 --opt-slice_timing-type_scanner Siemens --opt-slice_timing-type_acquisition interleaved ascending --opt-slice_timing-delay_in_tr 0 --opt-resample_vol-voxel_size 1 --opt-t1_preprocess-nu_correct-arg -distance 50 --opt-time_filter-hp 0.01 --opt-time_filter-lp Inf --opt-regress_confounds-flag_gsc true --opt-regress_confounds-flag_scrubbing true --opt-regress_confounds-thre_fd 0.5 --opt-smooth_vol-fwhm 6 --subjects 1-1570 --opt-corsica-threshold 0.15 --opt-corsica-flag_skip false --opt-size_output quality_control --opt-motion_correction-suppress_vol 3 --opt-granularity max

I am having problems with docker on guillimin. Now that singulrity 2.2 is installed everywhere, it might be time to switch. Also, got a email from Briarée and they are looking into installing singularity too.

• Keeps the inputs from the 13.5 version even if I switch to the beta version.
• Always tells me that the output entry is not a string, I then have to rewrite it, resubmit, and hope that it will be accepted as a string.

Various enhancements discussed so far (uncategorized and unprioritized):

• Think of how PSOM workers could be distributed to multiple sites (PSOM is quite hard on the file system).
• Task scheduling and worker submission.
• Limit IO footprint by grouping tasks in workers.
• Does it make sense to use Boutiques to wrap command-line tools in Niak? Still to be discussed in Niak, but definitely in the web platform built around Niak.
• Push for Docker support in more sites.

• The "subjects to process" param in FmriPreprocess.json could be a little bit more documented. In particular, the expected format of the string should appear with an example.

Documentation of the other params could be reviewed too.

• Add a walltime parameter to the pre-processing pipeline. Defaults to 3 hours.
• Define a walltime_estimate (or whatever it is called in CBRAIN) method in https://github.com/SIMEXP/cbrain-plugins-psom/blob/master/cbrain_task/psom_pipeline_task/bourreau/psom_pipeline_task.rb that returns the value of this parameter.

There is a Boutiques property for that:

• walltime-estimate: Estimated wall time of a task, in seconds.

However, in the case of PSOM, it doesn't seem to be easy to set any walltime estimate as it depends on the number of subjects that will be processed. I guess a very conservative estimate (e.g. a few days) should be use on the master process to avoid it being killed, and workers could have a lower value to reduce their queuing time (e.g. 24 hours).

The output folder can be pretty big (more than 600 GB for the 1570 GSP subjects), we should only sync the report folder on the cbrain portal.

Users should be able to enter subject-specific parameters to a pipeline execution. After discussion with @pbellec and @poquirion, this will be implemented by:

• extending PSOM so that it can load a "fine-tuning" parameter configuration file that contains all the subject-specific params.
• adding this parameter file as an input of the pipelines in CBRAIN.
• add a validation tool to check parameter files before pipelines are launched. This validation tool could be integrated in the same way as the BIDS validator. It would allow users to tag invalid parameter files.