simexp / niak Goto Github PK

niak_glm does not add an intercept to the model provided by the user. If the user does not include an intercept it would be useful to issue a warning.

in the niak_pipeline_stability_fir pipeline, if you enter a baseline or condition name different from the csv model , niak fir estimation will be stuck and the jobs drops from the queue. It should drop an error message , or warning message before starting the pipeline.

They could also be disable altogether if it is possible.

It would make sense to have a separate docker image for Ubuntu+octave+minc, that can be used for development (adding local git images of niak and friends), and base the niak image on that.

opt.type_fdr = 'local' seems to work fine for glm_fir but not glm_connectome (see below).
At this stage, it is unclear for me whether it actually provides meaningful results even for glm_fir (does it provide an error message if it is ignored?)

Something went bad ... the job has FAILED !
The last error message occured was :
local is an unknown procedure to control the FDR

Using niak_pipeline_glm_connectome, when specifying an intrarun model for a subset of the runs (such as when trying to remove the effect of block design on one run and them compairing that run with a resting-state run), niak fails to compute the connectome of the runs.

Check that niak_coord_vox2world and niak_coord_world2vox also work in nifti.

There should be a tutorial documenting all steps for release, including archive preparation, testing, and upload.

Currently the scores pipeline operates on a structure where each input is stored as a string in a field with the filename. It should rather expect a structure where there is a hierarchy of fields for (in that order) subject, session and run. For example like the one generated by niak_grab_fmri_preprocess.

I need a short test dataset and template output for testing of new edits and for later integration into the travis framework. Could be a first testrun for the new github file repo.

the version name for targets should probably go by date. This way it will be easy to trace major changes to niak pipelines over time. Names for targets should be something like

target_test_niak_mnc1-YYYY-MM-DD.zip

To keep the alphabetical order flowing....

The current installation page only give the full recipe for local install. The default should be for people to use docker, as it ensures reproducible results.
https://registry.hub.docker.com/u/simexp/niak/

The description of the BASC pipeline in the documentation is outdated:
http://niak.simexp-lab.org/pipe_basc.html#basc-pipeline-publication-guidelines

It is also lacking the description of MSTEPS.

Right now, the fmri preprocessing pipeline crashes when the T1 - BOLD registration fails. There should maybe be a warning, but no error. The user should catch the problem through quality control, not through a cryptic error message and a failed job near the end of the pipeline.

When I am running the scores pipeline on ABIDE, I have several files failing - after being reset a couple of times. I checked some error logs and they all said something like this

Something went bad ... the job has FAILED !                                                                                         
The last error message occured was :                                                                                                
niak:read: . There was a problem unzipping the file. Please check that the command 'gunzip -f' works, or change this command using the variable GB_NIAK_UNZIP in the file NIAK_GB_VARS

File /sb/home/surchs/code/niak/commands/read_write/niak_read_vol.m at line 236
File /sb/home/surchs/code/niak/extensions/surfstab/niak_brick_scores_fmri.m at line 161
File /sb/home/surchs/code/psom/psom_run_job.m at line 214               
File /sb/home/surchs/code/psom/psom_run_job.m at line 128

I then tried to open these the file with fsl and also tried loading it with niak_read_vol in octave - both worked. Not sure what is happening since about 1/3 of the files seems affected.

One example on peuplier would be /media/database3/abide/abide_release_sym_gsc0_lp01/USM/fmri_0050528_session_1_run1.nii.gz

There is still an option to keep the fMRI data in native space in the template of the fMRI preprocessing pipeline. This is deprecated and should be removed.

With graphviz missing, it's not possible to show dependency graphs using PSOM.

Crawling a mnc folder with no further options specified breaks with following error

Error using niak_brick_mnc2nii (line 133)
mnc2nii: symbol lookup error: /usr/lib/x86_64-linux-gnu/libminc2.so.3.0.0:
undefined symbol: ncerr

Same thing in octave runs no problem

When adding the tests for scores to niak_test_all, we ran into a case where some differences existed between the source and the target, but the test job did not fail. The test is implemented like in many other pipelines, and the tests have worked in the past for these, so it's not clear what the issue may be.

Releasing a new version of NIAK currently includes:

• Generate a target for tests using niak_test_all with opt.flag_target = true.
• Zip the output folder target under the name target_test_niak_mnc1-xxxx.zip, where xxxx is the new version number (e.g. 0.13.0).
• Release the target on http://www.nitrc.org/frs/?group_id=411 in the test_niak package.
• Change the variable gb_niak_version in the file niak_gb_vars (line 88).
• Create a new tag on the local git repo git tag -a vxxxx
• Push the new tag git push --tags
• Edit the release page at https://github.com/SIMEXP/niak/releases/edit/vxxxx and describe the main changes with the previous release.
• Create a new build script for docker.
• build and push a new docker image, called simexp/niak:xxxx . Also push the same image under the tag simexp/niak:xxxx simexp/niak:latest
• push the niak images to docker (docker push niak:xxxx docker push niak:latest)
• start bash from the docker image, zip the /home/niak folder under the name niak-boss-xxxx.zip.
• release the zip on nitrc as part of the niak package.

In addition, to deploy on a server:

• wget the niak zip archive from nitrc
• copy psom_gb_vars_local.m from a previous release
• copy commands/misc/niak_gb_vars.m from a previous release.

This is awfully complicated but, fortunately, can be simplified a lot.

This seems like a pretty good candidate.

After inspecting the travis logs, the difference starts in the regression of confounds, and the difference with the target are tiny. After checking the faulty commit it turns out that the formula for regression in niak_glm was modified for a more numerically stable method which likely explains the change. We should simply ignore this difference and release a new target.

niak_write_vol does apparently not check whether the dimensions in the header under hdr.details.dim match the actual data dimensions. As a result, you can save 3D data with a 4D header without any warnings or prompts.

Loading the same data with niak_read_vol() however throws an error because the data dimensions don't match the header.

The current implementation of the nifti reader/writer is not tested/robust enough. It would be better to simply write a wrapper around a well tested tool, such as Jimmy Chen's libraries.

A couple suggestions, more to come:

• the input format is not properly documented. But it doesn't matter. I suggest only supporting cell of strings in the brick, and only the structure fmri.(subject).(session).(run) in the pipeline. The mapping between the two will occur in the pipeline.
• I would get rid of the .mat option. First the outputs are not properly documented. Second it makes things way too complicated. I will incorporate the remapping of 4D volumes into .mat files as part of the subtype analysis.
• the checks lines 171 to 182 are ugly and very error prone. In any case I don't think it is necessary to get a warning for that. I suggest removing them altogether.
• the if statement line 306 is redundant with lines 307 & 317. Please remove. There is a series of similar if statement.
• Overall the code is too long. consider making sub-functions.

It may be interesting to have max_ttest maps in addition du perc_disc maps as a systematic output of the GLM_FIR analysis. This is not urgent though, as such maps can be easily generated based on the already available test maps...

niak_test_all now generates some weird verbose:

I have discovered the following file ending: .mnc.gz
I have discovered the following file ending: .mnc.gz

This needs to be removed.

the vector perc_discovery does not match what is found in the percentage of discovery volumes in the GLM-FIR pipeline. There is clearly a bug. A good thing this pipeline is at an experimental stage.

Connectivity pipelines fail because they depend on functions in bct (brain connectivity tool box) folder. Did you chose to not include it by purpose in the latest release?

currently, to run the test suite it is necessary to download several archives, and run several lines of code. It would be better by default to have the test suite automatically download all necessary data (if absent) and run the test in the current directory.

we need to build a virtual machine that run linux OS and with NIAK intalled and tested.

It would be useful to allow users to specify which template they want to use. This involves the specification of a bunch of different files:

• a T1 template
• a brain mask
• a head mask
• a dilated mask
• an eroded mask
• a mask of the pure white matter
• a mask of the ventricles (conservative)
• a mask of the circle of Willis

Needs to be done.

We need an option to specify a set of connections that we want to do a FDR local correction on it.

global variables in niak_gb_vars should use the values defined in PSOM, when psom is present.

We recently decided that the output of scores would take the form of a .mat file with several variable networkNN where NN=1,... would be the number of the network, and would contain the corresponding stability maps. It would be useful to have the map vectorized, i.e. the output of niak_vol2tseries, and not the full 3D volume. This is save a lot of disk&memory space. I would then be important to have two other variables present: mask and hdr. mask is the binary brain mask used to vectorize the maps. hdr if the header of the 3D volume.

csv are not encoded the same way for dos based platform and unix, in dos the standard \r is replace with a ^M in the text. the function niak_read_csv and niak_read_csv_cell should be able to handle this.

I'm processing data for a study in which subject IDs can contain dashes and/or underscores, which Niak doesn't like.

Could Niak be made to digest a broader range of ID formats?

The text in "" is not handle correctly if we have commas in the string they should be ignore

Hi,

not sure if this only affects me but psom_visu_dependencies fails for me on the docker image with the following error:

sh: 1: dot: not found
warning: I could not find the BIOGRAPH command or the DOT command. This probably means that the Matlab bioinformatics toolbox is not installed and that you did not install the GRAPHVIZ package on your system. Sorry dude, I can't plot the graph.

I also noticed that I don't get the Octave gui from the image so maybe something is wrong with my system. I still labeled it as enhancement for now in case it really doesn't work.

changes in the late stages of the preprocessing pipeline should not restart from raw data. Somehow currently it is the case, despite intermediate outputs being kept.

niak test on Matlab in magma.
Created niak branch called "niak_matlab" , fixed a minor compatibility bug and pushed the branch to github. je piepline stopped after 31 fail.

path result on magma: /scratch/pbellec/yassinebha/results
path target on magma: /scratch/pbellec/yassinebha/target_niak-boss-0.12.14/

the log message for the fiirst failed job fp_sica_subject1_run1 :

Spatial independent component analysis

Reading data /scratch/pbellec/yassinebha/results/demoniak_preproc/intermediate/subject1/regress_confounds//fmri_subject1_session1_motor_cor.mnc.gz ...
Reading brain mask /scratch/pbellec/yassinebha/results/demoniak_preproc/anat/subject1/func_subject1_mask_stereonl.mnc.gz ...
Correction of the mean of time series ...
Performing spatial independent component analysis with 60 components, this might take a while ...

Something went bad ... the job has FAILED !
The last error message occured was :
Error using inv
Too many output arguments.
File /scratch/pbellec/yassinebha/niak/commands/sica/niak_sica.m at line 943
File /scratch/pbellec/yassinebha/niak/commands/sica/niak_sica.m at line 183
File /scratch/pbellec/yassinebha/niak/bricks/sica/niak_brick_sica.m at line 238
File /scratch/pbellec/yassinebha/psom/psom_run_job.m at line 214
File /scratch/pbellec/yassinebha/psom/psom_run_job.m at line 128

The installation for non-linux users is confusing ... it would be great to have a section for non-linux users that says:

• install virtualbox
• install the niak virtual machine

It would be more logical to rename the pipelines as follows:

• basc_connectome, basc_fir
• glm_connectome, glm_fir

This implies also modifying the code used for testing.

The pipeline generates faulty headers for the following output types:

• stability_contrast
• stability_inter
• stability_intra

Files can be read with fsl and afni but not with nibabel or niak. Exact cause is still unclear, probably has something to do with the way I am reusing template headers to save the outputs.

If a file niak_gb_vars_local is detected, it should override niak_gb_vars. This is already implented for psom_gb_vars in the PSOM project.

Hi, Pierre.

The demo-script problem seems to still persist with regard to the local temporary folder. I have attached the issue from last time as well as this time for documentation purposes.

Issue Thursday:

Issue this time, after sourcing from quarantine:

Issue this time, after sourcing directly from latest pulled GitHub fork (as of 2014-07-17 22:50):

testing niak-boss-0.12.14 on mammouth.
path_test.demoniak = '/mnt/parallel_scratch_ms2_wipe_on_december_2014/pbellec/pbellec_group/niak_test/data_demo';
path_test.target = '/mnt/parallel_scratch_ms2_wipe_on_december_2014/pbellec/pbellec_group/niak_test/target_niak-boss-0.12.14';
path_test.result = '/mnt/parallel_scratch_ms2_wipe_on_december_2014/pbellec/pbellec_group/niak_test/test_result/';

the test completed with 5 jobd fail:
fp_test_regression ;
cc_graph_prop_rois_subject1 ;
cc_graph_prop_rois_subject2 ; rest_test_stability_rest_demoniak ;
glmconn_test_glm_conntome_demoniak ;

The current test suite is slow (way over an hour) and generates about 1 Gb of data. It would be very useful to have a lightweight test suite. For example, it may be feasible to down sample by 2 or even three both the functional and anatomical data (at native and stereotaxic resolution). This should bring the computation time and output size down to something reasonable.