Code Monkey home page Code Monkey logo

ciftify's Introduction

CircleCIBinder

DOI

ciftify

The tools of the Human Connectome Project (HCP) adapted for working with non-HCP datasets

ciftify is a set of three types of command line tools:

  1. conversion tools : command line tools adapted from HCP Minimal processing pipeline to put preprocessed T1 and fMRI data into an HCP like folder structure
  2. ciftify tools : Command line tools for making working with cifty format a little easier
  3. cifti_vis tools : Visualization tools, these use connectome-workbench tools to create pngs of standard views the present theme together in fRML pages.

Check out our wiki for more details on individual tools!

https://edickie.github.io/ciftify/

Download and Install

Install latest release python package

First, install the python package and all of its bundled data and scripts. You can do this with a single command with pip.

Note the newest release of ciftify requires python 3 (python 2 no longer supported).

To install with pip, type the following in a terminal.

pip install ciftify

For other installation options see this installation documentation.

ciftify workflows

Scripts adapted from HCP Minimal processing pipeline to put preprocessed T1 and fMRI data into an HCP like folder structure

  • ciftify_recon_all
    • Will convert any freeserfer output directory into an HCP (cifti space) output directory
  • ciftify_subject_fmri
    • Will project a nifti functional scan to a cifti .dtseries.nii in that subjects hcp analysis directory
    • The subject's hcp analysis directory is created by runnning ciftify_recon_all on that participants freesurfer output
    • will do fancy outlier removal to optimize the mapping in the process and then smooth the data in cifti space

ciftify Tools

  • ciftify_meants:
    • extracts mean timeseries(es) (similar to FSL' fslmeants) that can take nifti, cifti or gifti inputs
  • ciftify_seed_corr:
    • builds seed-based correlation maps using cifti, gifti or nifti inputs
  • ciftify_peaktable:
    • similar to FSL's clusterize, outputs a csv table of peak locations from a cifti statisical map
  • ciftify_surface_rois:
    • a tool for building circular rois on the cortical surface. Multiple roi locations can be read at once from a csv table.
  • ciftify_groupmask:
    • a tools for building a group mask for statiscal analyses using multiple .dtseries.nii files as the input
  • cifity_vol_result
    • Will project a nifti scan to cifti space (4D nifti -> .dtseries.nii or 3D nifti -> .dsclar.nii) with no fancy steps or smoothing
    • intended for conversion of 3D statistical maps (or 3D regions of interest) for visualization with wb_view

cifti_vis Tools

  • citfi_vis_recon_all:
    • builds visual qc pages for verification of ciftify_recon_all conversion
    • Note: these pages can also be used for qc of freesurfer's recon-all pipeline
    • (they easier to generate (i.e. no display needed) than freesurfer QAtools, and a little prettier too)
  • cifti_vis_fmri:
    • builds visual qc pages for verification of ciftify_subject_fmri volume to surface mapping
    • also show surface smoothed images of seed based connectivity to give an impression of preprocessed fmri data quality
  • cifti_vis_map:
    • generates picture of standard views from any cifti map (combined into on .html page)
    • One can loop over multiple files (i.e. maps from multiple subjects) and combine all outputs so that all subjects can viewed together in one index page.
    • can also take a nifti input which is internally converted to cifti using ciftify_vol_result
  • cifti_vis_RSN:
    • From a functional file input, Will run seed-based correlations from 4 ROIS of interest then generate pics of standard views
    • One can loop over multiple files (i.e. maps from multiple subjects) and combine all outputs so that all subjects can viewed together in one index page.
    • can also take a nifti input which is internally converted to cifti using ciftify_vol_result

And also in the bin there is

These two are part of a work in progress (something I need to validate first) ciftify_PINT_vertices cifti_vis_PINT epi_hcpexport

References / Citing ciftify

The workflows and template files employed in ciftify were adapted from those of the Human Connectome Project's minimal proprocessing pipeline. As such, any work employing ciftify's conversion of visualization tools should cite:

Glasser MF, Sotiropoulos SN, Wilson JA, Coalson TS, Fischl B, Andersson JL, Xu J, Jbabdi S, Webster M, Polimeni JR, Van Essen DC, Jenkinson M, WU-Minn HCP Consortium. The minimal preprocessing pipelines for the Human Connectome Project. Neuroimage. 2013 Oct 15;80:105-24. PubMed PMID: 23668970; PubMed Central PMCID: PMC3720813.

Additionally, any work employing the parcellation files included here should cite their original sources. They are:

Yeo 7 or (17) Network Parcellation: Yeo, B. T. Thomas, Fenna M. Krienen, Jorge Sepulcre, Mert R. Sabuncu, Danial Lashkari, Marisa Hollinshead, Joshua L. Roffman, et al. 2011. “The Organization of the Human Cerebral Cortex Estimated by Intrinsic Functional Connectivity.” Journal of Neurophysiology 106 (3): 1125–65.

The freesurfer DK atlas (i.e. 'aparc' segmentation): Desikan, Rahul S., Florent Ségonne, Bruce Fischl, Brian T. Quinn, Bradford C. Dickerson, Deborah Blacker, Randy L. Buckner, et al. 2006. “An Automated Labeling System for Subdividing the Human Cerebral Cortex on MRI Scans into Gyral Based Regions of Interest.” NeuroImage 31 (3): 968–80.

The Glasser MMP1.0 Parcellation: Glasser, Matthew F., Timothy S. Coalson, Emma C. Robinson, Carl D. Hacker, John Harwell, Essa Yacoub, Kamil Ugurbil, et al. 2016. “A Multi-Modal Parcellation of Human Cerebral Cortex.” Nature 536 (7615): 171–78.

ciftify's People

Contributors

chrisgorgo avatar delaneyjohnston avatar dependabot[bot] avatar desm1th avatar edickie avatar jerdra avatar jooh avatar lgrennan avatar mmanogaran avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

ciftify's Issues

check on labels for

./ciftify_seed_corr/sub-50005_tvertex_func_ciftitocifti_unmasked_meants.csv (should it be there?)
./ciftify_seed_corr/sub-50005_tvertex7_giftitocifti_masked_cortex.csv (correctly named??)
./ciftify_seed_corr/sub-50005_tvertex7_giftitocifti_masked_subcortical.csv
(correctly named?)

Any special steps needed for MNI152NLin2009cAsym?

I preprocessed my data using fmriprep. I'd like to use the data that's transformed to the MNI Asymmetric template and been denoised with AROMA (e.g., *bold_space-MNI152NLin2009cAsym_variant-smoothAROMAnonaggr_preproc.nii.gz). For ciftify_subject_fmri I used the --already-in-MNI flag, but would I need to do anything differently in terms of processing the T1w? I'm assuming not since the ciftify_recon_all step just uses the freesurfer outputs, but I wanted to check.

Thanks!

change cifify_a_nifti name and default usage

In reponse to valid concerns that mapping to the S900 average surfaces is, most of the time, not the best decision. We are doing to change the usage so that this is not an easy thing to do..

also ciffity_a_nifti will get renamed ciffity_nifti_result to emplasize to the user that the intended usage is for the presentation of result from a previous volume smoothed analysis..

add standard logging to cifitfy-PINT-vertices

This probably needs to happen before we do the validation experiments in corr

I might be that when we tune the parameters..the number of needed iterations might change..

This really just mean moving all the print statements in cifitfy-PINT-veritices into a logger and logging to a file pint.log?

rename fs2hcp and func2hcp

I'm thinking
fs2hcp.py -> ciftify_recon_all
func2hcp.py -> ciftify_fmri_nifti

let's just drop all the links that don't have .py at the end?

add nilearn cleaning functionality

add functinality to use nilearn cleaning for filtering and confound regression of cifti files (and signals)

Note: we need to build in functionality that it smarter for memory usage (i.e. doesn't try to clean the while image as once if the image is too big..)

utilities run command not setting the right log level...

The ciftify_recon_all.log ciftify_subject_fmri.log files aren't don't contain the "Running: cmd" sections..

They do contains the section headers, which are also at the info level...and error messages... so maybe the run function is internally resetting the log level??

syntax error line 475 fs2hcp.py

 474         run(['wb_command', '-add-to-spec-file', os.path.realpath(spec_file(     
 475                 subject_id, mesh_settings)), 'INVALID', dlabel_file]) << note square bracket

bug in 2.0.3

I think there might be a bug in the latest update. I tried installed the update with pip but there was an error. (I replicated the problem on another computer with Python 3.6.5)

Successfully built ciftify
Installing collected packages: ciftify
Exception:
Traceback (most recent call last):
File "/apps/psyc/anaconda3/python35_520/lib/python3.6/site-packages/pip/_internal/basecommand.py", line 228, in main
status = self.run(options, args)
File "/apps/psyc/anaconda3/python35_520/lib/python3.6/site-packages/pip/_internal/commands/install.py", line 335, in run
use_user_site=options.use_user_site,
File "/apps/psyc/anaconda3/python35_520/lib/python3.6/site-packages/pip/_internal/req/init.py", line 49, in install_given_reqs
**kwargs
File "/apps/psyc/anaconda3/python35_520/lib/python3.6/site-packages/pip/_internal/req/req_install.py", line 748, in install
use_user_site=use_user_site, pycompile=pycompile,
File "/apps/psyc/anaconda3/python35_520/lib/python3.6/site-packages/pip/_internal/req/req_install.py", line 961, in move_wheel_files
warn_script_location=warn_script_location,
File "/apps/psyc/anaconda3/python35_520/lib/python3.6/site-packages/pip/_internal/wheel.py", line 327, in move_wheel_files
console, gui = get_entrypoints(ep_file)
File "/apps/psyc/anaconda3/python35_520/lib/python3.6/site-packages/pip/_internal/wheel.py", line 143, in get_entrypoints
console = dict(_split_ep(v) for v in console.values())
ValueError: dictionary update sequence element #4 has length 3; 2 is required

So I did a manual install and still have an issue running commands:

[joseph.orr@gfx bin$ ciftify_subject_fmri --SmoothingFWHM 2 --ciftify-work-dir $outdir --already-in-MNI $func $subject $label
Fatal Python error: Py_Initialize: can't initialize sys standard streams
Traceback (most recent call last):
File "/apps/psyc/ciftify/2.0.3/ciftify/io.py", line 9, in
File "/apps/psyc/anaconda3/python35_520/lib/python3.6/logging/init.py", line 26, in
File "/apps/psyc/anaconda3/python35_520/lib/python3.6/traceback.py", line 5, in
File "/apps/psyc/anaconda3/python35_520/lib/python3.6/linecache.py", line 11, in
File "/apps/psyc/anaconda3/python35_520/lib/python3.6/tokenize.py", line 27, in
ImportError: cannot import name 'open'
Aborted (core dumped)

How to run "ciftify usage example" in a docker

First of all, thank a lot to share your work.
I wish to run ciftify in a docker so I cloned from github the project ciftify, build the Dockerfile and ran it....
But I do not understand how that can work ....because It seems to me that the dockerfile open only a jupyter notebook and thats it ...
I think I have missed something....
Could you please say a word on that ?

ciftify_subject_fmri error

I tried to convert Nifti a 4D volume to surface and I got this error:

ciftify_subject_fmri MYBASEDIR/fmri_s0025447_session3_rest.nii.gz 0025447 out
Subject HCPfolder does not exit

Any insights related to this please ?

Error in ciftify_subject_fmri with --func-ref option

File "${CODEPATH}/ciftify/bin/ciftify_subject_fmri.py", line 522, in define_func_3D
cifify.meants.verify_nifti_dimensions_match(settings.ref_vol.path,
NameError: name 'cifify' is not defined

After correcting the typo, running into below issue

ciftify.meants.verify_nifti_dimensions_match(settings.ref_vol.path,
AttributeError: 'Settings' object has no attribute 'ref_vol'

Here is the command that is used

ciftify_subject_fmri --surf-reg FS --ciftify-work-dir ${PROCESSDIR} --FLIRT-to-T1w --func-ref ${DATAPATH}/fmri_map1.nii.gz fmri_mapAll.nii.gz SUBJECT mapname

Please assist if you see any issue in the usage

Thanks

integrate other Erin's QC views

This is a QC scene for the HCP Pipelines

  • integrate the views (especially MSMSulc & Myelin related ones)
  • retest that the scene export works..as a func and other??
    + default location for the scene should be within the subjects hcp folder? i.e. same level as T1w and MNINonlinear folders..

separate the cifti images for smoothing

There is some weird (memory leak?) thing happening with wb_command -cifti-smoothing where everything freezes if you try to smooth more than one image in parallel on the same machine... Will switch to separating and smoothing the three parts (instead of the cifti file) to see if this goes away.

deprecate the --hcp-data-dir flag

We got some worry that the --hcp-data-dir flag is not well named (it's not acutally data from the HCP) so we will rename it (and the enviroment variable) and add deprecation warnings to the current version

-deprecation warning if the flag is used
-deprecation warning in the help docs
-help docs will switch to using new flag

New flag is:
--ciftify-working-dir?
--ciftify-data-dir?

log error typo thingy ciftify_clean_img

In ciftify_clean_img

logger.error('Indicated colBIDS_ZHH/sub-10186/ses-02/func/sub-10186_ses-02_task-rest_bold.nii.gzumn {} not in confounds'.format(colname_arg))

Don't imaging all that ZHH bit is supposed to be coded in there : )

File exists error

Hi, when I run ciftify_recon_all script in the newest version, some errors appear. By checking the path, I find two MNINonLinear folds generated.

/root/projects/ciftify/ciftify/output/sub-50005/T1w/Native
/tmp/tmpzrV5el/T1w/native
/root/projects/ciftify/ciftify/output/sub-50005/MNINonLinear/Native
/tmp/tmpzrV5el/MNINonLinear/native
_/root/projects/ciftify/ciftify/output/sub-50005/MNINonLinear
_

Traceback (most recent call last):
File "/usr/local/bin/ciftify_recon_all", line 11, in
load_entry_point('ciftify==2.0.2', 'console_scripts', 'ciftify_recon_all')()
File "/usr/local/lib/python2.7/site-packages/ciftify-2.0.2-py2.7.egg/ciftify/bin/ciftify_recon_all.py", line 1436, in main
run_ciftify_recon_all(tmpdir, settings)
File "/usr/local/lib/python2.7/site-packages/ciftify-2.0.2-py2.7.egg/ciftify/bin/ciftify_recon_all.py", line 106, in run_ciftify_recon_all
os.path.join(subject.atlas_space_dir, 'Results'))
File "/usr/local/lib/python2.7/site-packages/ciftify-2.0.2-py2.7.egg/ciftify/bin/ciftify_recon_all.py", line 400, in create_output_directories
ciftify.utils.make_dir(mesh['Folder'], DRYRUN, suppress_exists_error = True)
File "/usr/local/lib/python2.7/site-packages/ciftify-2.0.2-py2.7.egg/ciftify/utils.py", line 73, in make_dir
os.makedirs(dir_name)
File "/usr/local/lib/python2.7/os.py", line 157, in makedirs
mkdir(name, mode)
OSError: [Errno 17] File exists: '/root/projects/ciftify/ciftify/output/sub-50005/MNINonLinear'

=================================================================================

I find some clues in ciftify_recon_all.py in line 398, create_output_directories function. The meshes.values include the following information:
{'ROI': 'roi', 'meshname': 'native', 'T2wImage': '/root/projects/ciftify/ciftify/output/sub-50005/T1w/T2w.nii.gz', 'DenseMapsFolder': '/root/projects/ciftify/ciftify/output/sub-50005/MNINonLinear/Native', 'T1wImage': '/root/projects/ciftify/ciftify/output/sub-50005/T1w/T1w.nii.gz', 'Folder': '/root/projects/ciftify/ciftify/output/sub-50005/T1w/Native', 'tmpdir': '/tmp/tmps9u7P2/T1w/native'}

{'ROI': 'roi', 'meshname': 'native', 'T2wImage': '/root/projects/ciftify/ciftify/output/sub-50005/MNINonLinear/T2w.nii.gz', 'T1wImage': '/root/projects/ciftify/ciftify/output/sub-50005/MNINonLinear/T1w.nii.gz', 'Folder': '/root/projects/ciftify/ciftify/output/sub-50005/MNINonLinear/Native', 'tmpdir': '/tmp/tmps9u7P2/MNINonLinear/native'}

{'ROI': 'atlasroi', 'meshname': '164k_fs_LR', 'T2wImage': '/root/projects/ciftify/ciftify/output/sub-50005/MNINonLinear/T2w.nii.gz', 'T1wImage': '/root/projects/ciftify/ciftify/output/sub-50005/MNINonLinear/T1w.nii.gz', 'Folder': '/root/projects/ciftify/ciftify/output/sub-50005/MNINonLinear', 'tmpdir': '/tmp/tmps9u7P2/164k_fs_LR'}

Add more to clusterize

Things to add
[X ] volume calculation (i.e. cluster size)
[ ] atlas roi reading
[ X] subcortical component..

make space-T1w to MNI transform more faithful

The current plan, which we think works,

The problem: fmriprep and ciftify space-T1w nifti images are beautifully aligned (when you open them in more viewers) but they have different underlying sforms. This means that you can't just reslice one to the other using FLIRT (ident.mat) OR concatenate the ident.mat with the ciftify MNI transform to atlas transform the functionals from FMRIPREP.

Solution, (I think) is to use nilearn's nilearn.image.resample_to_image to do the reslicing (this seems to work/not be affected by the underlying different sforms). To get a FLIRT .mat file that also does this, I register the FMRIPREP func file to the nilearn resliced func file...

Will build this as a default behaviour of ciftitify_subject_fmri...also will switch the reg target to be the first image rather than the mean image.

Issue running citify_subject_fmri with --FLIRT-to-T1w option

citify_subject_fmri with --FLIRT-to-T1w as below is giving usage error

ciftify_subject_fmri --hcp-data-dir ${HCP_DIR} --FLIRT-to-T1w func.nii.gz subjid mapname

Error: Usage:
ciftify_subject_fmri [options] <func.nii.gz>

Its running fine without --FLIRT-to-T1w option as below,

ciftify_subject_fmri --hcp-data-dir ${HCP_DIR} func.nii.gz subjid mapname

Can you please suggest if you see any issues

make cifti_vis error out gracefully when dependancies not met

cifti vis requires workbench 1.2.3 and image magik to function properly..
When these are not loaded, it produces user-unfriendly errors.

Solution, add extra dependancy checks directly into the program..and user-friendly error reporting

Might consider doing this for all of the functions..

update documentition with license issues for dependancies

We are adding MSM as a dependancy.

Note this from T. Coalson:

Just a reminder: the MSM code/binary is forbidden from commercial use (similar to FSL), because of the terms of the ELC and HOCR discrete optimization libraries. Commercial interests would need to make arrangements with Waseda University (and maybe oxford, as it uses some FSL code, and I am unsure of the details of the FSL commercial restrictions).

I similar situation exists for Freesurfer and FSL.

Will need to add something to the docs about this.

adjust func2hcp behaviour to match current hcp pipelines..

From Tim Coalson

The func2hcp.py script uses the -volume-parcel-resampling-generic command that we took out of our pipelines, in order to instead do cubic spline resampling, resolving an issue of it creating a grid pattern in the noise variance (since this command uses a form of linear resampling). We now use -cifti-resample after making a subject-specific subcortical cifti file at the acquired resolution. See here:

https://github.com/Washington-University/Pipelines/blob/master/fMRISurface/scripts/SubcorticalProcessing.sh

find pint returning 0 in ivertex meants

Example:
"~/Documents/abide_PINT/20160908_PINT_fixa_UPSM_Long_Yeo7-80v_rad6-6-12/50007_followup_2/50007_followup_2_Yeo7-80v_rad6-6-12_ivertex_meants.csv"

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.