muscbridge / pydesigner Goto Github PK

A hands-free DTI, DKI, FBI and FBWM preprocessing pipeline. Information on algorithms and preprocessing steps are available at https://www.biorxiv.org/content/10.1101/2021.10.20.465189v1 A video tutorial on PyDesigner and its usage is now available at https://www.youtube.com/watch?v=mChQFuQqX3k

Home Page: https://pydesigner.readthedocs.io/en/latest/

License: Other

MATLAB 0.10% Python 97.35% Jupyter Notebook 2.40% Shell 0.08% Dockerfile 0.07%

dmri dmriprep diffusion-mri diffusion-maps dwi kurtosis diffusion-preprocessing tractography

pydesigner's Introduction

PyDesigner

GitHub release (latest SemVer including pre-releases)

Welcome to the official PyDesigner project!

PyDesigner was inspired by NYU's DESIGNER dMRI preprocessing pipeline to bring pre- and post- processing to every MRI imaging scientist. With PyDesigner, users are no longer confined to specific file types, operating systems, or complicated scripts just to extract DTI or DKI parameters – PyDesigner makes this easy, and you will love it!

PyDesigner Walkthrough Video: Software Setup & Usage

Notable Features

100% Python-based scripts
Minimized package dependencies for small package footprint
Preprocessing designed to boost SNR
Accurate and fast DTI and DKI metrics via cutting-edge algorithms
One-shot preprocessing to parameter extraction
Cross-platform compatibility between Windows, Mac and Linux using Docker
Highly flexible and easy to use
Parallel processing for quicker preprocessing and parameterization
Easy install with pip
Input file-format agnostic – works with .nii, .nii.gz, .mif and dicoms
Quality control metrics to evaluate data integrity – SNR graphs, outlier voxels, and head motion
Uses the latest techniques from DTI/DKI/FBI literature
Works with DTI, DKI, WMTI, FBI, or FBWM datasets
Supports multi-TE dataset processing
Tractography ready: Computes ODF spherical harmonic expansion for MRtrix3, and .fib files for DSI Studio
Installable modules for Python or Jupyter Notebook scripting of custom workflows

We welcome all DTI/DKI researchers to evaluate this software and pass on their feedback or issues through the Issues and Discussion page of this project’s GitHub repository.

System Requirements

Parallel processing in PyDesigner scales almost linearly with the nummber of CPU cores present. The application is also memory-intensive due to the number of parameter maps being computed.

Based on this evaluation, for processing a single DWI using PyDesigner, we recommend the following minimum system specifications:

Ubuntu 18.04
Intel i7-9700 or AMD Ryzen 1800X [8 cores]
16 GB RAM
12 GB free storage
Nvidia CUDA-enabled GPU

PyDesigner in Container

PyDesigner can run on all major platforms (x86, x64, and ARM) via Docker container technology. The containerized form of PyDesigner is called NeuroDock and comes preloaded with PyDesigner, FSL, and MRtrix3.

Obtaining the FSL software typically requires registration. Therefore, we requeust users of NeuroDock who have not previously registered as a user of the FSL software complete the registration process. This allows developers of FSL to be recognized for the utilization of their software.

Cite PyDesigner

Please include the following citation if you used PyDesigner in your work or publication:

Siddhartha Dhiman, Joshua B Teves, Kathryn E Thorn, Emilie T McKinnon, Hunter G Moss, Vitria Adisetiyo, Benjamin Ades-Aron, Jelle Veraart, Jenny Chen, Els Fieremans, Andreana Benitez, Joseph A Helpern, Jens H Jensen. PyDesigner: A Pythonic Implementation of the DESIGNER Pipeline for Diffusion Tensor and Diffusional Kurtosis Imaging. bioRxiv 2021.10.20.465189. doi: 10.1101/2021.10.20.465189

References

The PyDesigner software packages is based upon the the references listed below. Please be sure to cite them if PyDesigner was used in any publications.

Jensen JH, Helpern JA, Ramani A, Lu H, Kaczynski K. Diffusional kurtosis imaging: the quantification of non-Gaussian water diffusion by means of MRI. Magn Reson Med 2005;53:1432-1440. doi: 10.1002/mrm.20508
Jensen JH, Helpern JA. MRI Quantification of non-Gaussian water diffusion by kurtosis analysis. NMR Biomed 2010;23:698-710. doi: 10.1002/nbm.1518
Fieremans E, Jensen JH, Helpern JA. White matter characterization with diffusional kurtosis imaging. Neuroimage 2011;58:177-188. doi: 10.1016/j.neuroimage.2011.06.006
Tabesh A, Jensen JH, Ardekani BA, Helpern JA. Estimation of tensors and tensor-derived measures in diffusional kurtosis imaging. Magn Reson Med 2011;65:823-836. doi: 10.1002/mrm.22655
Glenn GR, Helpern JA, Tabesh A, Jensen JH. Quantitative assessment of diffusional kurtosis anisotropy. NMR Biomed 2015;28:448-459. doi: 10.1002/nbm.3271
Jensen JH, Glenn GR, Helpern JA. Fiber ball imaging. Neuroimage 2016; 124:824-833. doi: 10.1016/j.neuroimage.2015.09.049
McKinnon ET, Helpern JA, Jensen JH. Modeling white matter microstructure with fiber ball imaging. Neuroimage 2018;176:11-21. doi: 10.1016/j.neuroimage.2018.04.025
Ades-Aron B, Veraart J, Kochunov P, McGuire S, Sherman P, Kellner E, Novikov DS, Fieremans E. Evaluation of the accuracy and precision of the diffusion parameter EStImation with Gibbs and NoisE removal pipeline. Neuroimage. 2018;183:532-543. doi: 10.1016/j.neuroimage.2018.07.066
Moss H, McKinnon ET, Glenn GR, Helpern JA, Jensen JH. Optimization of data acquisition and analysis for fiber ball imaging. Neuroimage 2019;200;690-703. doi: 10.1016/j.neuroimage.2019.07.005
Moss HG, Jensen JH. Optimized rectification of fiber orientation density function. Magn Reson Med. 2020 Jul 25. doi: 10.1002/mrm.28406. Online ahead of print.

pydesigner's People

Contributors

Stargazers

Watchers

Forkers

thejaeger ryn-thorn j-tseng tmspvn joyies muscbridgelab fuyingji102

pydesigner's Issues

Add .mif support to DWIFile valid choices

We should add .mif support to the DWIFile utility.

CSF masking doesn't work

Describe the Issue
The flag --csfmask does nothing because it has not been implemented yet. We need to add this in as soon as possible for smoothing and possibly more computations in the future.

Issue Reproduction

Supplemental Information

dwipreproc error with --undistort & --rpe_none options

Describe the Issue
Dataset: 2 DKI averages, 1 extra set of B0 averages
Goal: Run PyDesigner with --undisort and --rpe_non options in order to register/align all my averages prior to DKI map calculation to improve map quality
Issue: Unable to process dataset and get dwipreproc error (see attached jpg)

Issue Reproduction
Dataset available in shared Box: All Files/PyDesigner_Testing/dwipreproc_Error/4042
Command ran: $ python /Users/adisetiyo/Desktop/PyDesigner/Software/PyDesigner/designer/pydesigner.py --denoise --undistort --smooth --rician --mask --rpe_none --verbose -o /Users/adisetiyo/Desktop/PyDesigner/4042/DKI_outputs /Users/adisetiyo/Desktop/PyDesigner/4042/DKI_inputs/4042_DIFF_siemens_dir_27_AVE1.nii,/Users/adisetiyo/Desktop/PyDesigner/4042/DKI_inputs/4042_DIFF_siemens_dir_27_AVE2.nii,/Users/adisetiyo/Desktop/PyDesigner/4042/DKI_inputs/4042_DKI_B0.nii

Supplemental Information
OS: macOS Mojave Version 10.14.5
See attached snapshots for 1) Error; 2) System set up

Dependencies are not tracked

designer.py does not track different program dependencies. As an example, a user could run without dwidenoise and then call rician correction, but the program will not detect this dependency unavailability and crash at the rician step.

Deprecate CVXOPT

Use CVXPY in its favor because it's easier to build and install.

Need setup.py

In order to create an executable that is easily ported into docker and other systems, we need a setup.py file which is properly configured.

Flag to keep/delete intermediary files

Need to implement flag to keep or remove intermediary preprocessing files. If keeping, create appropriate preprocessing folder in output directory and move them there. If not, write nothing but metrics.

DT and KT tensor order is incorrect

@andrew-taylor-2 performed fiber tracking with our tensorReordering() function and discovered that the order is incorrect. We will have to double check our outputs.

IRLLS Outlier Detection gets Singular Matrix error

When running dataset 5001 with the following command
designer -o ../des/ --denoise --degibbs --rician --mask DIFF_siemens_dir_2_7_AVE1
I get the following error:
numpy.linalg.LinAlgError: Singular matrix
Full traceback:

Traceback (most recent call last):
  File "/home/jbteves/PyDesigner/designer/pydesigner.py", line 635, in <module>
    outliers, dt_est = img.irlls()
  File "/home/jbteves/PyDesigner/designer/fitting/dwipi.py", line 1384, in irlls
    (delayed(outlierHelper)(dwi[:, i], bmat, sigma[i,0], b, b0_pos) for i in inputs))
  File "/home/jbteves/miniconda3/envs/designerdev/lib/python3.7/site-packages/joblib/parallel.py", line 934, in __call__
    self.retrieve()
  File "/home/jbteves/miniconda3/envs/designerdev/lib/python3.7/site-packages/joblib/parallel.py", line 833, in retrieve
    self._output.extend(job.get(timeout=self.timeout))
  File "/home/jbteves/miniconda3/envs/designerdev/lib/python3.7/site-packages/joblib/_parallel_backends.py", line 521, in wrap_future_result
    return future.result(timeout=timeout)
  File "/home/jbteves/miniconda3/envs/designerdev/lib/python3.7/concurrent/futures/_base.py", line 428, in result
    return self.__get_result()
  File "/home/jbteves/miniconda3/envs/designerdev/lib/python3.7/concurrent/futures/_base.py", line 384, in __get_result
    raise self._exception

Topup doesn't get denoised

Describe the Issue
Topup input does not get same preprocessing treatment as input DWI. Having a corrected topup series may possibly produce better maps

Issue Reproduction

Supplemental Information

Cannot force run `--degibbs`

I have a manually concatenated DWI with a .json file that does not contain Fourier information. I do know, however, that this DWI has full Fourier. Here's the error produced:

Traceback (most recent call last):
  File "PyDesigner.py", line 307, in <module>
    if args.degibbs and filetable['dwi'].isPartialFourier():
  File "/Users/sid/Repos/PyDesigner/designer/preprocessing/util.py", line 300, in isPartialFourier
    encoding = self.json['PartialFourier']
KeyError: 'PartialFourier'

We need to implement an --adv flag for advanced users to bypass checks and force a preprocessing step. This would add another if after L310.

if args.degibbs and args.adv:
    args.degibbs = True

Eddy doesn't run with RPE None

Users have the option of performing eddy motion correction only without topup with the flag -rpe_none. While not needed for motion correction, this still requires a phase encoding (PE) direction. Current implementation for motion correction only works with the flag -eddy -rpe-none -pe_dir j, where j is the PE direction.

It would be nice to not specify a PE direction if no topup is being run such that the above flasg becomes -eddy -rpe_none. Be wary though as this may be an MRTRIX3 limitation rather than designer itself.

Need to auto-detect or guide full/partial fourier encoding

Need to know if full or partial fourier encoding was used for mrdegibbs to work properly.

Divide by zeros in tensor fitting

There are currently two places in tensor fitting where divide-by-zero could possibly occur. This produces lots of warnings (unless disabled) and possibly inaccurate kurtosis parameters.

The first is in FA calculation of dwipi.py. In particular

The second occurs in AKC calculation when ADC = 0, in the function kurtosisCoeff at this line

Here's a log of terminal output from one such erroneous runs for reference


```/Users/sid/Downloads/PyDesigner_Test/In/preprocessed_dwi.nii
Image preprocessed_dwi.nii loaded successfully
IRLLS: Noise Estimation : 100%|█| 187039/187039 [00:08<00:00, 23170.22vox/s]
IRLLS: Outlier Detection: 100%|█| 187039/187039 [00:59<00:00, 3140.20vox/s]
Constrained Tensor Fit  :   5%| | 9873/187039 [00:02<00:38, 4633.91vox/s]/Users/sid/miniconda3/envs/deep/lib/python3.7/site-packages/joblib/externals/loky/process_executor.py:706: UserWarning: A worker stopped while some jobs were given to the executor. This can be caused by a too short worker timeout or by a memory leak.
  "timeout or by a memory leak.", UserWarning
Constrained Tensor Fit  : 100%|█| 187039/187039 [00:46<00:00, 4034.20vox/s]
Detected DKI
DTI params              : 100%|█| 187039/187039 [00:13<00:00, 13905.56vox/s]
/Users/sid/Repos/PyDesigner/designer/fitting/dwipi.py:663: RuntimeWarning: divide by zero encountered in true_divide
  l1 ** 2 + l2 ** 2 + l3 ** 2)
/Users/sid/Repos/PyDesigner/designer/fitting/dwipi.py:663: RuntimeWarning: invalid value encountered in true_divide
  l1 ** 2 + l2 ** 2 + l3 ** 2)
AKC Outlier Detection   : 100%|██████| 10/10 [00:23<00:00,  2.34s/blk]
AKC Correction          : 100%|███| 21/21 [00:10<00:00,  2.07tensor/s]
/Users/sid/Repos/PyDesigner/designer/fitting/dwipi.py:338: RuntimeWarning: divide by zero encountered in true_divide
  akc = (akc * np.tile(md**2, (adc.shape[0], 1)))/(adc**2)
/Users/sid/Repos/PyDesigner/designer/fitting/dwipi.py:338: RuntimeWarning: invalid value encountered in true_divide
  akc = (akc * np.tile(md**2, (adc.shape[0], 1)))/(adc**2)
DKI params              :   1%| | 2523/187039 [00:00<06:33, 468.83vox/s]/Users/sid/Repos/PyDesigner/designer/fitting/dwipi.py:338: RuntimeWarning: invalid value encountered in true_divide
  akc = (akc * np.tile(md**2, (adc.shape[0], 1)))/(adc**2)
/Users/sid/Repos/PyDesigner/designer/fitting/dwipi.py:338: RuntimeWarning: divide by zero encountered in true_divide
  akc = (akc * np.tile(md**2, (adc.shape[0], 1)))/(adc**2)
DKI params              :   2%| | 3491/187039 [00:00<04:40, 654.43vox/s]/Users/sid/Repos/PyDesigner/designer/fitting/dwipi.py:338: RuntimeWarning: invalid value encountered in true_divide
  akc = (akc * np.tile(md**2, (adc.shape[0], 1)))/(adc**2)
/Users/sid/Repos/PyDesigner/designer/fitting/dwipi.py:338: RuntimeWarning: divide by zero encountered in true_divide
  akc = (akc * np.tile(md**2, (adc.shape[0], 1)))/(adc**2)
/Users/sid/Repos/PyDesigner/designer/fitting/dwipi.py:338: RuntimeWarning: invalid value encountered in true_divide
  akc = (akc * np.tile(md**2, (adc.shape[0], 1)))/(adc**2)
/Users/sid/Repos/PyDesigner/designer/fitting/dwipi.py:338: RuntimeWarning: invalid value encountered in true_divide
  akc = (akc * np.tile(md**2, (adc.shape[0], 1)))/(adc**2)
/Users/sid/Repos/PyDesigner/designer/fitting/dwipi.py:338: RuntimeWarning: divide by zero encountered in true_divide
  akc = (akc * np.tile(md**2, (adc.shape[0], 1)))/(adc**2)
/Users/sid/Repos/PyDesigner/designer/fitting/dwipi.py:338: RuntimeWarning: divide by zero encountered in true_divide
  akc = (akc * np.tile(md**2, (adc.shape[0], 1)))/(adc**2)
DKI params              :   3%| | 5219/187039 [00:01<03:17, 918.52vox/s]/Users/sid/Repos/PyDesigner/designer/fitting/dwipi.py:338: RuntimeWarning: invalid value encountered in true_divide
  akc = (akc * np.tile(md**2, (adc.shape[0], 1)))/(adc**2)
/Users/sid/Repos/PyDesigner/designer/fitting/dwipi.py:338: RuntimeWarning: divide by zero encountered in true_divide
  akc = (akc * np.tile(md**2, (adc.shape[0], 1)))/(adc**2)
DKI params              :   4%| | 7811/187039 [00:01<01:44, 1713.52vox/s]/Users/sid/Repos/PyDesigner/designer/fitting/dwipi.py:338: RuntimeWarning: invalid value encountered in true_divide
  akc = (akc * np.tile(md**2, (adc.shape[0], 1)))/(adc**2)
/Users/sid/Repos/PyDesigner/designer/fitting/dwipi.py:338: RuntimeWarning: divide by zero encountered in true_divide
  akc = (akc * np.tile(md**2, (adc.shape[0], 1)))/(adc**2)
DKI params              :   5%| | 9780/187039 [00:01<01:19, 2219.63vox/s]/Users/sid/Repos/PyDesigner/designer/fitting/dwipi.py:338: RuntimeWarning: invalid value encountered in true_divide
  akc = (akc * np.tile(md**2, (adc.shape[0], 1)))/(adc**2)
/Users/sid/Repos/PyDesigner/designer/fitting/dwipi.py:338: RuntimeWarning: divide by zero encountered in true_divide
  akc = (akc * np.tile(md**2, (adc.shape[0], 1)))/(adc**2)
DKI params              :   7%| | 12707/187039 [00:02<00:57, 3016.26vox/s]/Users/sid/Repos/PyDesigner/designer/fitting/dwipi.py:338: RuntimeWarning: invalid value encountered in true_divide
  akc = (akc * np.tile(md**2, (adc.shape[0], 1)))/(adc**2)
/Users/sid/Repos/PyDesigner/designer/fitting/dwipi.py:338: RuntimeWarning: divide by zero encountered in true_divide
  akc = (akc * np.tile(md**2, (adc.shape[0], 1)))/(adc**2)
DKI params              :   8%| | 15615/187039 [00:02<00:34, 4934.08vox/s]/Users/sid/Repos/PyDesigner/designer/fitting/dwipi.py:338: RuntimeWarning: invalid value encountered in true_divide
  akc = (akc * np.tile(md**2, (adc.shape[0], 1)))/(adc**2)
/Users/sid/Repos/PyDesigner/designer/fitting/dwipi.py:338: RuntimeWarning: divide by zero encountered in true_divide
  akc = (akc * np.tile(md**2, (adc.shape[0], 1)))/(adc**2)
DKI params              :  18%|▏| 32743/187039 [00:04<00:17, 8623.43vox/s]/Users/sid/Repos/PyDesigner/designer/fitting/dwipi.py:338: RuntimeWarning: invalid value encountered in true_divide
  akc = (akc * np.tile(md**2, (adc.shape[0], 1)))/(adc**2)
DKI params              :  18%|▏| 33875/187039 [00:04<00:17, 8741.16vox/s]/Users/sid/Repos/PyDesigner/designer/fitting/dwipi.py:338: RuntimeWarning: divide by zero encountered in true_divide
  akc = (akc * np.tile(md**2, (adc.shape[0], 1)))/(adc**2)
DKI params              :  26%|▎| 48995/187039 [00:06<00:16, 8456.37vox/s]/Users/sid/Repos/PyDesigner/designer/fitting/dwipi.py:338: RuntimeWarning: invalid value encountered in true_divide
  akc = (akc * np.tile(md**2, (adc.shape[0], 1)))/(adc**2)
/Users/sid/Repos/PyDesigner/designer/fitting/dwipi.py:338: RuntimeWarning: divide by zero encountered in true_divide
  akc = (akc * np.tile(md**2, (adc.shape[0], 1)))/(adc**2)
DKI params              :  30%|▎| 56771/187039 [00:07<00:16, 7683.50vox/s]/Users/sid/Repos/PyDesigner/designer/fitting/dwipi.py:338: RuntimeWarning: invalid value encountered in true_divide
  akc = (akc * np.tile(md**2, (adc.shape[0], 1)))/(adc**2)
/Users/sid/Repos/PyDesigner/designer/fitting/dwipi.py:338: RuntimeWarning: divide by zero encountered in true_divide
  akc = (akc * np.tile(md**2, (adc.shape[0], 1)))/(adc**2)
DKI params              :  32%|▎| 59651/187039 [00:07<00:15, 7990.62vox/s]/Users/sid/Repos/PyDesigner/designer/fitting/dwipi.py:338: RuntimeWarning: invalid value encountered in true_divide
  akc = (akc * np.tile(md**2, (adc.shape[0], 1)))/(adc**2)
/Users/sid/Repos/PyDesigner/designer/fitting/dwipi.py:338: RuntimeWarning: divide by zero encountered in true_divide
  akc = (akc * np.tile(md**2, (adc.shape[0], 1)))/(adc**2)
DKI params              :  40%|▍| 75635/187039 [00:09<00:12, 8624.48vox/s]/Users/sid/Repos/PyDesigner/designer/fitting/dwipi.py:338: RuntimeWarning: invalid value encountered in true_divide
  akc = (akc * np.tile(md**2, (adc.shape[0], 1)))/(adc**2)
/Users/sid/Repos/PyDesigner/designer/fitting/dwipi.py:338: RuntimeWarning: divide by zero encountered in true_divide
  akc = (akc * np.tile(md**2, (adc.shape[0], 1)))/(adc**2)
DKI params              :  44%|▍| 82118/187039 [00:10<00:12, 8479.08vox/s]/Users/sid/Repos/PyDesigner/designer/fitting/dwipi.py:338: RuntimeWarning: invalid value encountered in true_divide
  akc = (akc * np.tile(md**2, (adc.shape[0], 1)))/(adc**2)
/Users/sid/Repos/PyDesigner/designer/fitting/dwipi.py:338: RuntimeWarning: divide by zero encountered in true_divide
  akc = (akc * np.tile(md**2, (adc.shape[0], 1)))/(adc**2)
DKI params              :  45%|▍| 84995/187039 [00:10<00:12, 8306.00vox/s]/Users/sid/Repos/PyDesigner/designer/fitting/dwipi.py:338: RuntimeWarning: invalid value encountered in true_divide
  akc = (akc * np.tile(md**2, (adc.shape[0], 1)))/(adc**2)
/Users/sid/Repos/PyDesigner/designer/fitting/dwipi.py:338: RuntimeWarning: divide by zero encountered in true_divide
  akc = (akc * np.tile(md**2, (adc.shape[0], 1)))/(adc**2)
DKI params              :  48%|▍| 89747/187039 [00:11<00:11, 8500.36vox/s]/Users/sid/Repos/PyDesigner/designer/fitting/dwipi.py:338: RuntimeWarning: invalid value encountered in true_divide
  akc = (akc * np.tile(md**2, (adc.shape[0], 1)))/(adc**2)
/Users/sid/Repos/PyDesigner/designer/fitting/dwipi.py:338: RuntimeWarning: divide by zero encountered in true_divide
  akc = (akc * np.tile(md**2, (adc.shape[0], 1)))/(adc**2)
DKI params              :  48%|▍| 90611/187039 [00:11<00:11, 8490.68vox/s]/Users/sid/Repos/PyDesigner/designer/fitting/dwipi.py:338: RuntimeWarning: invalid value encountered in true_divide
  akc = (akc * np.tile(md**2, (adc.shape[0], 1)))/(adc**2)
/Users/sid/Repos/PyDesigner/designer/fitting/dwipi.py:338: RuntimeWarning: divide by zero encountered in true_divide
  akc = (akc * np.tile(md**2, (adc.shape[0], 1)))/(adc**2)
DKI params              :  57%|▌| 106739/187039 [00:13<00:09, 8541.98vox/s]/Users/sid/Repos/PyDesigner/designer/fitting/dwipi.py:338: RuntimeWarning: invalid value encountered in true_divide
  akc = (akc * np.tile(md**2, (adc.shape[0], 1)))/(adc**2)
/Users/sid/Repos/PyDesigner/designer/fitting/dwipi.py:338: RuntimeWarning: divide by zero encountered in true_divide
  akc = (akc * np.tile(md**2, (adc.shape[0], 1)))/(adc**2)
DKI params              : 100%|█| 187039/187039 [00:22<00:00, 8151.34vox/s]

Add dcm2niix SOP

Need to add this documentation prior to external testing.

Median filter too rough, change to mean filter

According to @TheJaeger median filter is too rough and should be changed to mean. Even though median filtering looks fine in MK, it may mess up FA since "bad" is different for each diffusion metric.

Kurtosis tensors appear to be more sensitive to outliers in DT.

Force flag doesn't run rician correction

--force won't run rician correction and returns the error:

$ python pydesigner.py --denoise --degibbs --smooth --rician --force --verbose file.nii -o outfolder
Given DWI is partial fourier, overriding --degibbs; no unringing correction will be done to avoid artifacts.
dwidenoise: [WARNING] existing output files will be overwritten
dwidenoise: [100%] preloading data for "/Users/kayti/Desktop/Projects/DESIGNER/KTGFDesignerTest/Data/5001/nifti/DIFF_siemens_dir_2_7_AVE2.nii"
dwidenoise: [100%] running MP-PCA denoising
Traceback (most recent call last):
File "/Users/kayti/Repos/PyDesigner/designer/pydesigner.py", line 492, in
raise Exception('Running rician correction would cause an '
Exception: Running rician correction would cause an overwrite. In order to run this please delete the files, use --force, use --resume, or change output destination.

Make issue template

We should make an issue template for users to submit their issues, including directions on searching the issue tracker to see if the issue is already known.

Documentation: Installation Edits

NEED:

Explicitly state if there are version dependencies for FSL, MRtrix, (if not, state which versions have tested and confirmed run on, like with Python 3.7).
Explicitly state if having older versions or duplicate versions on OS cause issues.
Include examples of how to check which versions of FSL, MRtrix, Python on computer:
Terminal $ python -V
Terminal $ fsleyes (Top Tab: FSLeyes/About FSLeyes)
Terminal $ mrconvert -version
FSL 'Advanced Users Only' section - confusing and unclear if this step is required for CUDA systems (plese rewrite and explicity state if section requried for CUDA systems or just optional)

Originally posted by @vadisetiyo in #50 (comment)

Index 22 out of bounds error

Receiving the following error across multiple data sets, subjects, and sequences:

Path: /Users/kayti/Desktop/Projects/DESIGNER/KTGFDesignerTest/Data/5001/designer/DIFF_siemens_dir_2_7_AVE1
Name: rsdDIFF_siemens_dir_2_7_AVE1
Full: /Users/kayti/Desktop/Projects/DESIGNER/KTGFDesignerTest/Data/5001/designer/DIFF_siemens_dir_2_7_AVE1/rsdDIFF_siemens_dir_2_7_AVE1.nii
No brain mask supplied
Image preprocessed_dwi.nii loaded successfully
Processing with 16 workers...
IRLLS: Noise Estimation : 100%|█| 329476/329476 [00:14<00:00, 22298.82vox/s]
IRLLS: Outlier Detection: 100%|█| 329476/329476 [01:17<00:00, 4261.73vox/s]
Constrained Tensor Fit  :   0%|           | 0/329476 [00:00<?, ?vox/s]Traceback (most recent call last):
  File "/Users/kayti/Repos/PyDesigner-Develop/designer/pydesigner.py", line 552, in <module>
    img.fit(fit_constraints, reject=outliers, dt_hat=dt_est)
  File "/Users/kayti/Repos/PyDesigner-Develop/designer/fitting/dwipi.py", line 568, in fit
    for i in inputs)
  File "/Users/kayti/anaconda3/lib/python3.7/site-packages/joblib/parallel.py", line 924, in __call__
    while self.dispatch_one_batch(iterator):
  File "/Users/kayti/anaconda3/lib/python3.7/site-packages/joblib/parallel.py", line 754, in dispatch_one_batch
    self._pickle_cache)
  File "/Users/kayti/anaconda3/lib/python3.7/site-packages/joblib/parallel.py", line 210, in __init__
    self.items = list(iterator_slice)
  File "/Users/kayti/Repos/PyDesigner-Develop/designer/fitting/dwipi.py", line 568, in <genexpr>
    for i in inputs)
IndexError: index 22 is out of bounds for axis 1 with size 22
^CError in atexit._run_exitfuncs:
Traceback (most recent call last):
  File "/Users/kayti/anaconda3/lib/python3.7/multiprocessing/util.py", line 265, in _run_finalizers
    finalizer()
  File "/Users/kayti/anaconda3/lib/python3.7/multiprocessing/util.py", line 189, in __call__
    res = self._callback(*self._args, **self._kwargs)
  File "/Users/kayti/anaconda3/lib/python3.7/site-packages/joblib/externals/loky/process_executor.py", line 197, in _python_exit
    thread.join()
  File "/Users/kayti/anaconda3/lib/python3.7/threading.py", line 1032, in join
    self._wait_for_tstate_lock()
  File "/Users/kayti/anaconda3/lib/python3.7/threading.py", line 1048, in _wait_for_tstate_lock
    elif lock.acquire(block, timeout):
KeyboardInterrupt

/Users/kayti/anaconda3/lib/python3.7/site-packages/joblib/externals/loky/backend/semaphore_tracker.py:198: UserWarning: semaphore_tracker: There appear to be 22 leaked semaphores to clean up at shutdown
  len(cache))

Need to integrate new CLI

Need to merge new CLI interface into the develop branch, which adds options for each supported feature in a clean manner.

Use threadpoolctl to control thread count

See the same issue I opened for tedana.

Incorporate a working.mif

Describe the Issue
MRTRIX3's tools work better with .mif files so create a working.mif instead that gets converted to .nii at the end of each preprocessing step.

Issue Reproduction

Supplemental Information

Add number of workers option

Add flag that allows users to define number of cores to use in processing. Default should be all available cores.

Use the multiprocessing library to manage thread detection:

import multiprocessing
num_cores = multiprocessing.cpu_count()

Add verbose provenance

Describe the Issue
Need a way to keep track of all preprocessing steps done so far in individual file headers: .jsonand .mif. This is to avoid ambiguity.

Add SNR plot

The original designer supplies an SNR plot for QC, which we should replicate in Python.

Allow adjustable FWHM for smoothing

Describe the Issue
Smoothing FWHM is permanently 1.2 unless we adjust the code (hard coded).

Issue Reproduction

Supplemental Information

Add WMTI

We should add WMTI to the pipeline.

`outlier` flag should call IRLLS; if not call LLS

The outlier flag should determine the fitting method used.

Write eddy_qc output into metrics_qc folder

Describe the Issue
Like the title says, we need to consolidate QC metrics into a single folder to keep the output folder neat and organized.

Add dicom support

The latest develop only supports .nii/.json pairs. We should add dicom support such that a user can supply either a .zip or a directory of dicoms and they will be converted before kicking off the pipeline. We will base this off of the original designer dicom conversion here. This will also require adding .mif support as described in #33

Alternative calculation to determine partial fourier

Some datasets do not have the PartialFourierdicom or json field, however this can be inferred based on other information that is almost always present in header. If NumberofPhaseEncodingSteps < AcquisitionMatrix[0], acquisition is PartialFourier=True.

Unable to use --rpe_pair and --pe_dir

Describe the Issue
--rpe_pair and --pe_dir does not work when trying to do eddy and topup

Issue Reproduction
Command:
python pydesigner.py \
--undistort --rpe_pair --pe_dir AP
--nofit --verbose \
--force -o $path/pydesigner/undistort
$input1,$input2

Error:
Cannot undistort without rpe selection

Supplemental Information

AttributeError: 'NoneType' object has no attribute 'shape'

$ python pydesigner.py --denoise --degibbs --smooth --rician file.nii -o outfolder
Given DWI is partial fourier, overriding --degibbs; no unringing correction will be done to avoid artifacts.
Path: /Users/kayti/Desktop/Projects/DESIGNER/KTGFDesignerTest/Data/5001/designer/DIFF_siemens_dir_2_7_AVE2
Name: rsdDIFF_siemens_dir_2_7_AVE2
Full: /Users/kayti/Desktop/Projects/DESIGNER/KTGFDesignerTest/Data/5001/designer/DIFF_siemens_dir_2_7_AVE2/rsdDIFF_siemens_dir_2_7_AVE2.nii
No brain mask supplied
Image preprocessed_dwi.nii loaded successfully
Processing with 16 workers...
IRLLS: Noise Estimation : 100%|█| 329476/329476 [00:15<00:00, 21219.59vox/s]
IRLLS: Outlier Detection: 100%|█| 329476/329476 [01:24<00:00, 3907.48vox/s]
Constrained Tensor Fit : 100%|█| 329476/329476 [04:31<00:00, 1214.13vox/s]
DTI params : 100%|█| 329476/329476 [00:22<00:00, 14779.67vox/s]
Traceback (most recent call last):
File "/Users/kayti/Repos/PyDesigner/designer/pydesigner.py", line 560, in
dp.writeNii(exec(f), img.hdr, fpath)
File "/Users/kayti/Repos/PyDesigner/designer/fitting/dwipi.py", line 1803, in writeNii
clipped_img = nib.Nifti1Image(map, hdr.affine, hdr.header)
File "/Users/kayti/anaconda3/lib/python3.7/site-packages/nibabel/nifti1.py", line 1774, in init
file_map)
File "/Users/kayti/anaconda3/lib/python3.7/site-packages/nibabel/analyze.py", line 923, in init
dataobj, affine, header, extra, file_map)
File "/Users/kayti/anaconda3/lib/python3.7/site-packages/nibabel/spatialimages.py", line 468, in init
self.update_header()
File "/Users/kayti/anaconda3/lib/python3.7/site-packages/nibabel/nifti1.py", line 2046, in update_header
super(Nifti1Image, self).update_header()
File "/Users/kayti/anaconda3/lib/python3.7/site-packages/nibabel/nifti1.py", line 1809, in update_header
super(Nifti1Pair, self).update_header()
File "/Users/kayti/anaconda3/lib/python3.7/site-packages/nibabel/spatialimages.py", line 490, in update_header
shape = self._dataobj.shape
AttributeError: 'NoneType' object has no attribute 'shape'

Rician correction does not account for `nan` in noise map

Describe the Issue
Rician correction does not account for nan in the noise map, which leads to signal destruction in those regions of DWI.

Issue Reproduction
Issue was caught when processing KTGF 4042. This data can be accessed from our shared data repository for testing.

Supplemental Information
Screenshot shows DWI before Rician, after Rician, and the output MK map.

Need CircleCI config

Need to configure CircleCI to test various functions.

Add DWI fitting parameters to CLI

We should add fitting parameters to the CLI. @TheJaeger please add a complete list of options to be added.

Naming must be exact

The program does not check what files have been generated and assumes certain filenames; this prevents users from inserting already preprocessed data to do tensor fitting.

Unrecognized arguments error when using rician and denoise

Describe the Issue
My inputs are unrecognized arguments when just using denoise and rician.

Issue Reproduction
Command:
python /Path/to/pydesigner.py
--denoise --rician --nofit
--verbose \
--force -o $path/pydesigner/rician
$input1,$input2

Error: error: unrecognized arguments: input1.nii,input2.nii

Supplemental Information

WMTI_AWF less than zero

Describe the Issue
There can be some voxels with a negative value close to zero. One such voxel was -1.5E-6 in a 2mm dataset. This will produce a complex number where AWF is involved in a sqrt, which shouldn't be the case because 0 <= AWF <= 1. One can resolve this with the line AWF[AWF < minZero] = minZero

Handling multi-volume inputs

PyDesigner needs a way to concatenate multiple volumes from different sequences, for studies that have multiple runs and b-values separated into different volumes. Refer to original DWI-Desginer for reference on dealing with this.

Our current approach of dealing with NifTi allows resuming, but makes it harder to concatenate volumes and headier information accompanying them. The approach used DKI-Designer did not allow resuming, but did concatenate volumes. A fine balance would likely have to be made somewhere, or an entirely different approach is needed.

Separate DTI and DKI parameter extraction

Current method forces computation of both types of parameters, which may fail if imaging protocol is DTI only. Additionally, median filtering of tensor affects DKI maps positively and DTI maps negatively. So we want to extract DTI maps prior prior to median filtering, then apply median filtering, followed by DKI map extraction.

Level 1 Testing - installation 101

Please include an installation 101 guide (link or text) prominantly on the intro page. Think anyone including your grandma can read and install Py-Designer starting from the PyDesigner home page.

Add logger

We need to configure a logger for the application so that users can get logfiles.

Begin file detection with `mrconvert`

We need to add mrconvert back again to detect file type. This would make it easier to auto-detect between DICOM and NifTi formats. In addition, there is a possibility we may add Bruker processing, so this little function will save us major headaches int he future.

DWI concatenation does not work with `--resume`

Describe the Issue
Concatenation of DWIs with multiple inputs fails when the --resume flag is parsed. Issue occurs because DWIParser is unable to overwrite the existing file.

Extract brain mask

Tensor estimation is up to ~3x faster if limited only within brain region. Extracting the brain mask right after undistort seems the best as all volumes are aligned. Take note that bet will fail if there are nans in the volume the brain mask is being extracted from, so fslmaths has to be used to remove these.

mrconvert -fslgrad <path_to_bvec> <path_to_bval> <path_to_eddy_out> <path_to_working.mif>
dwiextract -bzero <path_to_working.mif> - | mrmath -axis 3 - mean <path_to_b0.nii>
fslmaths <path_to_b0.nii> -nan <path_to_b0.nii>
bet <path_to_b0.nii> <path_to_brain_mask> -m -f <threshold>

L1: Converts undistorted DWI and its gradient table to .mif for simple b0 extraction
L2: Extracts all B0s in DWI and computes and pipes it into mrmath to determine mean, then saves the file as b0.nii.
L3: Converts nans to zeros. Keep this b0.nii in final output folder for integration with other pipelines.
L4: Computes brain mask. Default threshold should be 0.25 and users should be able to specify the threshold in CLI input parameter.

Add full vs half-sphere detection

Inject original designer's code into the new CLI. Original code here

Add KFA map

We should the KFA map as an output.

`--undistort` fails without DWI .json

Describe the Issue
Running --undistort --rpe_none on a subject without .json accompanying the DWI volume fails with the following traceback.

Traceback (most recent call last):
  File "pydesigner.py", line 468, in <module>
    preparation.make_simple_mif(filetable)
  File "/home/sid/Repos/PyDesigner/designer/preprocessing/preparation.py", line 57, in make_simple_mif
    if not op.exists(filetable['dwi'].getJSON()):
  File "/home/sid/miniconda3/envs/py3/lib/python3.7/genericpath.py", line 19, in exists
    os.stat(path)
TypeError: stat: path should be string, bytes, os.PathLike or integer, not NoneType
(py3) sid@sid-ubuntu:~/Repos/PyDesigner/designer$

Issue Reproduction
Run --undistort --rpe_none on a DWI without .json

Supplemental Information
It appears that the script is unable to convert to .mif without a .json file. Change the code such that it converts the file regardless, but users would have to manually specify --adv flag along with parameters such as --pe_dir j- to process the file with.

muscbridge / pydesigner Goto Github PK

pydesigner's Introduction

PyDesigner

Welcome to the official PyDesigner project!

Notable Features

PyDesigner in Container

Cite PyDesigner

References

pydesigner's People

Contributors

Stargazers

Watchers

Forkers

pydesigner's Issues

Recommend Projects

Recommend Topics

Recommend Org