penglu-washu / imc_denoise Goto Github PK
View Code? Open in Web Editor NEWIMC_Denoise: a software package to enhance Imaging Mass Cytometry - Nature Communications
License: Other
IMC_Denoise: a software package to enhance Imaging Mass Cytometry - Nature Communications
License: Other
Hi, so when i use the jupyter notebook and download the supplementary table 5, and it says the metal is not found and i can not run the demo. am I not pointing to the right directory?
The channel names follow the "144Nd-CD14_Nd144.tiff" format from the Zenodo deposit. is there a issue to fix this? is this a naming convention issue?
if 'generated_patches' in globals():
del generated_patches
channel_name = "144Nd"
Raw_directory = "C:/Users/antho/Documents/GithubRepos/supplemental data/IMC_Denoise/Raw_IMC_dataset_for_training_supp_table5/Raw_IMC_dataset/local-implementation-test" # change this directory to your Raw_image_directory.
Save_directory = None
n_neighbours = 4 # Larger n enables removing more consecutive hot pixels.
n_iter = 3 # Iteration number for DIMR
DataGenerator = DeepSNiF_DataGenerator(channel_name = channel_name, n_neighbours = n_neighbours, n_iter = n_iter)
generated_patches = DataGenerator.generate_patches_from_directory(load_directory = Raw_directory)
if DataGenerator.save_patches(generated_patches, save_directory = Save_directory):
print('Data generated successfully!')
Hi,
I am interested in using the DIMR and DeepSNiF algorithms to pre-process my IMC data. However, I have a M1 MacBook and can only use CPU locally to process the data and will not have access to an NVIDIA gpu. In this case, will I still be able to use the DIMR algorithm, and if there is an option to run DeepSNiF on a local gpu or cpu?
Have you tested the updated N2V2 architecture for IMC Denoise? Namely, using blurpool, removing the skip connections at top of U-Net, using non-residual blocks, and median instead of uniform_withCP for masking. Additionally, I noticed that IMC Denoise uses uniform_withoutCP instead of uniform_withCP as in the N2V repo. Is there a reason for this choice?
Hello!
I have started working with the package and it seems to be working well so far! It looks like it could be very helpful for my work.
I was wondering two things:
How should DeepSNiF typically be trained? For a given experiment, I had just been training fresh for each new experiment with patches generated from the entire experiment / all ROIs -- which means a lot of patches and can take a long time for a large experiment -- and then denoising the experiment. This seems not correct to me, although the output appears to be working well. Is it more typical to choose a subset of images from an experiment and train on those, then denoise the rest of the images from the experiment? Or is it more usual to train a single version of DeepSNiF, save the weights, and then re-use that trained model for multiple experiments?
I have been analyzing .mcd files and IMC images using the steinbock docker pipeline from the Bodenmiller lab, which produces a single folder with multi-channel .tiff files. This files structure, of course, does not work with how y'all originally designed with package. I have successfully modified the DeepSNiF_DataGenerator.py script and one of the example jupyter notebook file to be able to take in multi-channel tiffs housed in a single file folder for training (as steinbock outputs), then denoise selected channels, and finally export updated multi-channel tiffs with the selected channels replaced with the denoised output. Would you be interested adding this functionality into IMC_denoise? The edits required seem to be fairly small, and while my jupyter notebook script is only designed to do the whole-dataset-training-then-denoise method I have been employing, that is easily changed. I'd be happy to share my scripts / edits if you like!
Dear Peng,
We are keen to use IMC-Denoise for our IMC data. To get a feel for the tool we have downloaded and unzipped Raw_IMC_dataset_for_training_supp_table5.zip from https://doi.org/10.5281/zenodo.6533905.
IMC-Denoise installation was performed on Linux Mint 19.3 (NVIDIA GTX 3060), Ubuntu 20.04 (NVIDIA A40) and Debian 12 (NVIDIA GTX 3060 Ti) systems using the following commands:
conda create -n 'IMC' python=3.6
conda activate IMC
conda install -c anaconda brotlipy
conda install -c anaconda cudnn=7.6.5 cudatoolkit=10.1.243
pip install tensorflow==2.2.0 keras==2.3.1
pip install jupyter
Installation went without errors/problems on all systems. GPUs are visible in Python. We followed the Jupyter notebook IMC_Denoise_Train.ipynb. However, all 3 systems produce NaN loss values during training on the provided training data:
Epoch 1/50
124/124 [==============================] - 235s 2s/step - loss: nan - val_loss: nan
Epoch 00001: saving model to /home/thijs/trained_weights/weights_144Nd-CD14.hdf5
Epoch 2/50
124/124 [==============================] - 79s 634ms/step - loss: nan - val_loss: nan
Epoch 00002: saving model to /home/thijs/trained_weights/weights_144Nd-CD14.hdf5
Epoch 3/50
124/124 [==============================] - 78s 633ms/step - loss: nan - val_loss: nan
Epoch 00003: saving model to /home/thijs/trained_weights/weights_144Nd-CD14.hdf5
Epoch 4/50
124/124 [==============================] - 78s 632ms/step - loss: nan - val_loss: nan
Epoch 00004: saving model to /home/thijs/trained_weights/weights_144Nd-CD14.hdf5
Epoch 5/50
124/124 [==============================] - 78s 632ms/step - loss: nan - val_loss: nan
Epoch 00005: saving model to /home/thijs/trained_weights/weights_144Nd-CD14.hdf5
Epoch 6/50
124/124 [==============================] - 78s 632ms/step - loss: nan - val_loss: nan
Epoch 00006: saving model to /home/thijs/trained_weights/weights_144Nd-CD14.hdf5
Could you please advise how to proceed? Is it possible that there is a NaN value in the training data?
Hi,
I am interested in using IMC Denoise to preprocess my IMC data. Particularly, I plan applying the DIMR algorithm to remove hot pixels, but am still open to options on which approach to use for background noise removal (due to absence of an NVIDIA GPU at the moment). My question is whether it it technically logical to use DIMR for hot pixel removal but to couple it with another background noise removal strategy (ie. Ilastik binary classification)?
In addition, I was also wondering whether it is recommended to perform hot pixel filtration, background noise, and lastly spillover compensation in this order, regardless of the algorithms picked for data pre-processing.
Thank you so much for your help!
hi, so i have the DeNoise jupyter notebook running, i can generate the training images, and visualize the plots. however when i look at the loss functions, the NaN values are computed using the training data from the jupyter notebooks. is this correct? i can't reproduce the Github version ?
The range value to the corresponding model is 68.200004196167.
Input Channel Shape => (18608, 64, 64, 1)
Number of Training Examples: 15816
Number of Validation Examples: 2792
Each training patch with shape of (64, 64) will mask 8 pixels.
Training model...
Epoch 1/50
124/124 [==============================] - 909s 7s/step - loss: nan - val_loss: nan
Epoch 00001: saving model to C:\Users\antho\Documents\GithubRepos\IMC_Denoise\Jupyter_Notebook_examples\trained_weights\weights_144Nd-CD14.hdf5
Epoch 2/50
124/124 [==============================] - 72s 582ms/step - loss: nan - val_loss: nan
Epoch 00002: saving model to C:\Users\antho\Documents\GithubRepos\IMC_Denoise\Jupyter_Notebook_examples\trained_weights\weights_144Nd-CD14.hdf5
Epoch 3/50
124/124 [==============================] - 72s 581ms/step - loss: nan - val_loss: nan
Epoch 00003: saving model to C:\Users\antho\Documents\GithubRepos\IMC_Denoise\Jupyter_Notebook_examples\trained_weights\weights_144Nd-CD14.hdf5
Epoch 4/50
124/124 [==============================] - 72s 581ms/step - loss: nan - val_loss: nan
Epoch 00004: saving model to C:\Users\antho\Documents\GithubRepos\IMC_Denoise\Jupyter_Notebook_examples\trained_weights\weights_144Nd-CD14.hdf5
Epoch 5/50
124/124 [==============================] - 72s 582ms/step - loss: nan - val_loss: nan
Epoch 00005: saving model to C:\Users\antho\Documents\GithubRepos\IMC_Denoise\Jupyter_Notebook_examples\trained_weights\weights_144Nd-CD14.hdf5
Epoch 6/50
124/124 [==============================] - 72s 583ms/step - loss: nan - val_loss: nan
Epoch 00006: saving model to C:\Users\antho\Documents\GithubRepos\IMC_Denoise\Jupyter_Notebook_examples\trained_weights\weights_144Nd-CD14.hdf5
Epoch 7/50
124/124 [==============================] - 72s 583ms/step - loss: nan - val_loss: nan
Epoch 00007: saving model to C:\Users\antho\Documents\GithubRepos\IMC_Denoise\Jupyter_Notebook_examples\trained_weights\weights_144Nd-CD14.hdf5
Epoch 8/50
124/124 [==============================] - 72s 581ms/step - loss: nan - val_loss: nan
Epoch 00008: saving model to C:\Users\antho\Documents\GithubRepos\IMC_Denoise\Jupyter_Notebook_examples\trained_weights\weights_144Nd-CD14.hdf5
Epoch 9/50
124/124 [==============================] - 72s 581ms/step - loss: nan - val_loss: nan
Epoch 00009: saving model to C:\Users\antho\Documents\GithubRepos\IMC_Denoise\Jupyter_Notebook_examples\trained_weights\weights_144Nd-CD14.hdf5
Epoch 10/50
124/124 [==============================] - 72s 581ms/step - loss: nan - val_loss: nan
Epoch 00010: saving model to C:\Users\antho\Documents\GithubRepos\IMC_Denoise\Jupyter_Notebook_examples\trained_weights\weights_144Nd-CD14.hdf5
Epoch 11/50
124/124 [==============================] - 72s 582ms/step - loss: nan - val_loss: nan
Epoch 00011: saving model to C:\Users\antho\Documents\GithubRepos\IMC_Denoise\Jupyter_Notebook_examples\trained_weights\weights_144Nd-CD14.hdf5
Epoch 12/50
124/124 [==============================] - 72s 582ms/step - loss: nan - val_loss: nan
Epoch 00012: saving model to C:\Users\antho\Documents\GithubRepos\IMC_Denoise\Jupyter_Notebook_examples\trained_weights\weights_144Nd-CD14.hdf5
Epoch 13/50
124/124 [==============================] - 72s 583ms/step - loss: nan - val_loss: nan
Epoch 00013: saving model to C:\Users\antho\Documents\GithubRepos\IMC_Denoise\Jupyter_Notebook_examples\trained_weights\weights_144Nd-CD14.hdf5
Epoch 14/50
124/124 [==============================] - 72s 584ms/step - loss: nan - val_loss: nan
Epoch 00014: saving model to C:\Users\antho\Documents\GithubRepos\IMC_Denoise\Jupyter_Notebook_examples\trained_weights\weights_144Nd-CD14.hdf5
Epoch 15/50
124/124 [==============================] - 72s 583ms/step - loss: nan - val_loss: nan
Epoch 00015: saving model to C:\Users\antho\Documents\GithubRepos\IMC_Denoise\Jupyter_Notebook_examples\trained_weights\weights_144Nd-CD14.hdf5
Epoch 16/50
124/124 [==============================] - 72s 583ms/step - loss: nan - val_loss: nan
Epoch 00016: saving model to C:\Users\antho\Documents\GithubRepos\IMC_Denoise\Jupyter_Notebook_examples\trained_weights\weights_144Nd-CD14.hdf5
Epoch 17/50
124/124 [==============================] - 72s 584ms/step - loss: nan - val_loss: nan
Epoch 00017: saving model to C:\Users\antho\Documents\GithubRepos\IMC_Denoise\Jupyter_Notebook_examples\trained_weights\weights_144Nd-CD14.hdf5
Epoch 18/50
124/124 [==============================] - 72s 582ms/step - loss: nan - val_loss: nan
Epoch 00018: saving model to C:\Users\antho\Documents\GithubRepos\IMC_Denoise\Jupyter_Notebook_examples\trained_weights\weights_144Nd-CD14.hdf5
Epoch 19/50
124/124 [==============================] - 72s 582ms/step - loss: nan - val_loss: nan
Epoch 00019: saving model to C:\Users\antho\Documents\GithubRepos\IMC_Denoise\Jupyter_Notebook_examples\trained_weights\weights_144Nd-CD14.hdf5
Epoch 20/50
124/124 [==============================] - 72s 582ms/step - loss: nan - val_loss: nan
Epoch 00020: saving model to C:\Users\antho\Documents\GithubRepos\IMC_Denoise\Jupyter_Notebook_examples\trained_weights\weights_144Nd-CD14.hdf5
Epoch 00020: ReduceLROnPlateau reducing learning rate to 0.0005000000237487257.
Epoch 21/50
124/124 [==============================] - 72s 582ms/step - loss: nan - val_loss: nan
Epoch 00021: saving model to C:\Users\antho\Documents\GithubRepos\IMC_Denoise\Jupyter_Notebook_examples\trained_weights\weights_144Nd-CD14.hdf5
Epoch 22/50
124/124 [==============================] - 72s 582ms/step - loss: nan - val_loss: nan
Epoch 00022: saving model to C:\Users\antho\Documents\GithubRepos\IMC_Denoise\Jupyter_Notebook_examples\trained_weights\weights_144Nd-CD14.hdf5
Epoch 23/50
124/124 [==============================] - 72s 582ms/step - loss: nan - val_loss: nan
Epoch 00023: saving model to C:\Users\antho\Documents\GithubRepos\IMC_Denoise\Jupyter_Notebook_examples\trained_weights\weights_144Nd-CD14.hdf5
Epoch 24/50
124/124 [==============================] - 72s 582ms/step - loss: nan - val_loss: nan
Epoch 00024: saving model to C:\Users\antho\Documents\GithubRepos\IMC_Denoise\Jupyter_Notebook_examples\trained_weights\weights_144Nd-CD14.hdf5
Epoch 25/50
124/124 [==============================] - 74s 601ms/step - loss: nan - val_loss: nan
Epoch 00025: saving model to C:\Users\antho\Documents\GithubRepos\IMC_Denoise\Jupyter_Notebook_examples\trained_weights\weights_144Nd-CD14.hdf5
Epoch 26/50
124/124 [==============================] - 72s 582ms/step - loss: nan - val_loss: nan
Epoch 00026: saving model to C:\Users\antho\Documents\GithubRepos\IMC_Denoise\Jupyter_Notebook_examples\trained_weights\weights_144Nd-CD14.hdf5
Epoch 27/50
124/124 [==============================] - 73s 586ms/step - loss: nan - val_loss: nan
Epoch 00027: saving model to C:\Users\antho\Documents\GithubRepos\IMC_Denoise\Jupyter_Notebook_examples\trained_weights\weights_144Nd-CD14.hdf5
Epoch 28/50
124/124 [==============================] - 72s 582ms/step - loss: nan - val_loss: nan
Epoch 00028: saving model to C:\Users\antho\Documents\GithubRepos\IMC_Denoise\Jupyter_Notebook_examples\trained_weights\weights_144Nd-CD14.hdf5
Epoch 29/50
124/124 [==============================] - 72s 582ms/step - loss: nan - val_loss: nan
Epoch 00029: saving model to C:\Users\antho\Documents\GithubRepos\IMC_Denoise\Jupyter_Notebook_examples\trained_weights\weights_144Nd-CD14.hdf5
Epoch 30/50
124/124 [==============================] - 73s 588ms/step - loss: nan - val_loss: nan
Epoch 00030: saving model to C:\Users\antho\Documents\GithubRepos\IMC_Denoise\Jupyter_Notebook_examples\trained_weights\weights_144Nd-CD14.hdf5
Epoch 31/50
124/124 [==============================] - 73s 587ms/step - loss: nan - val_loss: nan
Epoch 00031: saving model to C:\Users\antho\Documents\GithubRepos\IMC_Denoise\Jupyter_Notebook_examples\trained_weights\weights_144Nd-CD14.hdf5
Epoch 32/50
124/124 [==============================] - 73s 588ms/step - loss: nan - val_loss: nan
Epoch 00032: saving model to C:\Users\antho\Documents\GithubRepos\IMC_Denoise\Jupyter_Notebook_examples\trained_weights\weights_144Nd-CD14.hdf5
Epoch 33/50
124/124 [==============================] - 73s 585ms/step - loss: nan - val_loss: nan
Epoch 00033: saving model to C:\Users\antho\Documents\GithubRepos\IMC_Denoise\Jupyter_Notebook_examples\trained_weights\weights_144Nd-CD14.hdf5
Epoch 34/50
124/124 [==============================] - 72s 583ms/step - loss: nan - val_loss: nan
Epoch 00034: saving model to C:\Users\antho\Documents\GithubRepos\IMC_Denoise\Jupyter_Notebook_examples\trained_weights\weights_144Nd-CD14.hdf5
Epoch 35/50
124/124 [==============================] - 72s 585ms/step - loss: nan - val_loss: nan
Epoch 00035: saving model to C:\Users\antho\Documents\GithubRepos\IMC_Denoise\Jupyter_Notebook_examples\trained_weights\weights_144Nd-CD14.hdf5
Epoch 36/50
124/124 [==============================] - 72s 584ms/step - loss: nan - val_loss: nan
Epoch 00036: saving model to C:\Users\antho\Documents\GithubRepos\IMC_Denoise\Jupyter_Notebook_examples\trained_weights\weights_144Nd-CD14.hdf5
Epoch 37/50
124/124 [==============================] - 72s 583ms/step - loss: nan - val_loss: nan
Epoch 00037: saving model to C:\Users\antho\Documents\GithubRepos\IMC_Denoise\Jupyter_Notebook_examples\trained_weights\weights_144Nd-CD14.hdf5
Epoch 38/50
124/124 [==============================] - 72s 583ms/step - loss: nan - val_loss: nan
Epoch 00038: saving model to C:\Users\antho\Documents\GithubRepos\IMC_Denoise\Jupyter_Notebook_examples\trained_weights\weights_144Nd-CD14.hdf5
Epoch 39/50
124/124 [==============================] - 72s 583ms/step - loss: nan - val_loss: nan
Epoch 00039: saving model to C:\Users\antho\Documents\GithubRepos\IMC_Denoise\Jupyter_Notebook_examples\trained_weights\weights_144Nd-CD14.hdf5
Epoch 40/50
124/124 [==============================] - 72s 583ms/step - loss: nan - val_loss: nan
Epoch 00040: saving model to C:\Users\antho\Documents\GithubRepos\IMC_Denoise\Jupyter_Notebook_examples\trained_weights\weights_144Nd-CD14.hdf5
Epoch 00040: ReduceLROnPlateau reducing learning rate to 0.0002500000118743628.
Epoch 41/50
124/124 [==============================] - 72s 582ms/step - loss: nan - val_loss: nan
Epoch 00041: saving model to C:\Users\antho\Documents\GithubRepos\IMC_Denoise\Jupyter_Notebook_examples\trained_weights\weights_144Nd-CD14.hdf5
Epoch 42/50
124/124 [==============================] - 72s 583ms/step - loss: nan - val_loss: nan
Epoch 00042: saving model to C:\Users\antho\Documents\GithubRepos\IMC_Denoise\Jupyter_Notebook_examples\trained_weights\weights_144Nd-CD14.hdf5
Epoch 43/50
124/124 [==============================] - 72s 583ms/step - loss: nan - val_loss: nan
Epoch 00043: saving model to C:\Users\antho\Documents\GithubRepos\IMC_Denoise\Jupyter_Notebook_examples\trained_weights\weights_144Nd-CD14.hdf5
Epoch 44/50
124/124 [==============================] - 72s 583ms/step - loss: nan - val_loss: nan
Epoch 00044: saving model to C:\Users\antho\Documents\GithubRepos\IMC_Denoise\Jupyter_Notebook_examples\trained_weights\weights_144Nd-CD14.hdf5
Epoch 45/50
124/124 [==============================] - 72s 582ms/step - loss: nan - val_loss: nan
Epoch 00045: saving model to C:\Users\antho\Documents\GithubRepos\IMC_Denoise\Jupyter_Notebook_examples\trained_weights\weights_144Nd-CD14.hdf5
Epoch 46/50
124/124 [==============================] - 72s 584ms/step - loss: nan - val_loss: nan
Epoch 00046: saving model to C:\Users\antho\Documents\GithubRepos\IMC_Denoise\Jupyter_Notebook_examples\trained_weights\weights_144Nd-CD14.hdf5
Epoch 47/50
124/124 [==============================] - 72s 584ms/step - loss: nan - val_loss: nan
Epoch 00047: saving model to C:\Users\antho\Documents\GithubRepos\IMC_Denoise\Jupyter_Notebook_examples\trained_weights\weights_144Nd-CD14.hdf5
Epoch 48/50
124/124 [==============================] - 72s 582ms/step - loss: nan - val_loss: nan
Hi and thanks for this exciting package! Im trying to implement and test IMC denoise to a huge IMC dataset I
m working on. How did generate the image files that are used for the example data?
Hi. I`m running a 4090 RTX card with the script, but it gives me na values during Deep SNIF training. Which settings can I adjust for getting a value?
I am trying to run IMC_Denoise with Steinbock Jupyter Notebook. Initially, I encountered NaNs so I updated the cudNN and cudatoolkit accordingly. This solved the problem, however the training was very slow on our HPC.
This is my environment:
(IMC_Denoise2) [hd_bm327@o06c02(Singularity) Jupyter_Notebook_examples]$ conda list
# packages in environment at /home/hd/hd_hd/hd_bm327/.conda/envs/IMC_Denoise2:
#
# Name Version Build Channel
_libgcc_mutex 0.1 conda_forge conda-forge
_openmp_mutex 4.5 2_gnu conda-forge
absl-py 1.4.0 pypi_0 pypi
argon2-cffi 21.3.0 pypi_0 pypi
argon2-cffi-bindings 21.2.0 pypi_0 pypi
astunparse 1.6.3 pypi_0 pypi
async-generator 1.10 pypi_0 pypi
attrs 22.2.0 pypi_0 pypi
backcall 0.2.0 pypi_0 pypi
bleach 4.1.0 pypi_0 pypi
brotlipy 0.7.0 py36h8f6f2f9_1001 conda-forge
ca-certificates 2024.3.11 h06a4308_0
cachetools 4.2.4 pypi_0 pypi
certifi 2021.5.30 py36h5fab9bb_0 conda-forge
cffi 1.14.6 py36hc120d54_0 conda-forge
charset-normalizer 2.0.12 pypi_0 pypi
comm 0.1.4 pypi_0 pypi
cuda-version 11.2 hb11dac2_3 conda-forge
cudatoolkit 11.2.2 hc23eb0c_13 conda-forge
cudnn 8.1.0.77 h90431f1_0 conda-forge
cycler 0.11.0 pypi_0 pypi
dataclasses 0.8 pypi_0 pypi
decorator 5.1.1 pypi_0 pypi
defusedxml 0.7.1 pypi_0 pypi
entrypoints 0.4 pypi_0 pypi
gast 0.3.3 pypi_0 pypi
google-auth 1.35.0 pypi_0 pypi
google-auth-oauthlib 0.4.6 pypi_0 pypi
google-pasta 0.2.0 pypi_0 pypi
grpcio 1.48.2 pypi_0 pypi
h5py 2.10.0 pypi_0 pypi
idna 3.6 pypi_0 pypi
imc-denoise 0.0.0 dev_0 <develop>
importlib-metadata 4.8.3 pypi_0 pypi
ipykernel 5.5.6 pypi_0 pypi
ipython 7.16.3 pypi_0 pypi
ipython-genutils 0.2.0 pypi_0 pypi
ipywidgets 7.8.1 pypi_0 pypi
jedi 0.17.2 pypi_0 pypi
jinja2 3.0.3 pypi_0 pypi
joblib 1.1.1 pypi_0 pypi
jsonschema 3.2.0 pypi_0 pypi
jupyter 1.0.0 pypi_0 pypi
jupyter-client 7.1.2 pypi_0 pypi
jupyter-console 6.4.3 pypi_0 pypi
jupyter-core 4.9.2 pypi_0 pypi
jupyterlab-pygments 0.1.2 pypi_0 pypi
jupyterlab-widgets 1.1.7 pypi_0 pypi
keras 2.3.1 pypi_0 pypi
keras-applications 1.0.8 pypi_0 pypi
keras-preprocessing 1.1.2 pypi_0 pypi
kiwisolver 1.3.1 pypi_0 pypi
ld_impl_linux-64 2.38 h1181459_1
libffi 3.3 he6710b0_2
libgcc-ng 13.2.0 h807b86a_5 conda-forge
libgomp 13.2.0 h807b86a_5 conda-forge
libstdcxx-ng 13.2.0 h7e041cc_5 conda-forge
libzlib 1.2.13 hd590300_5 conda-forge
markdown 3.3.7 pypi_0 pypi
markupsafe 2.0.1 pypi_0 pypi
matplotlib 3.3.4 pypi_0 pypi
mistune 0.8.4 pypi_0 pypi
nbclient 0.5.9 pypi_0 pypi
nbconvert 6.0.7 pypi_0 pypi
nbformat 5.1.3 pypi_0 pypi
ncurses 6.4 h6a678d5_0
nest-asyncio 1.6.0 pypi_0 pypi
notebook 6.4.10 pypi_0 pypi
numpy 1.19.5 pypi_0 pypi
oauthlib 3.2.2 pypi_0 pypi
openssl 1.1.1w hd590300_0 conda-forge
opt-einsum 3.3.0 pypi_0 pypi
packaging 21.3 pypi_0 pypi
pandocfilters 1.5.1 pypi_0 pypi
parso 0.7.1 pypi_0 pypi
pexpect 4.9.0 pypi_0 pypi
pickleshare 0.7.5 pypi_0 pypi
pillow 8.4.0 pypi_0 pypi
pip 21.2.2 py36h06a4308_0
prometheus-client 0.17.1 pypi_0 pypi
prompt-toolkit 3.0.36 pypi_0 pypi
protobuf 3.19.6 pypi_0 pypi
ptyprocess 0.7.0 pypi_0 pypi
pyasn1 0.5.1 pypi_0 pypi
pyasn1-modules 0.3.0 pypi_0 pypi
pycparser 2.21 pyhd8ed1ab_0 conda-forge
pygments 2.14.0 pypi_0 pypi
pyparsing 3.1.2 pypi_0 pypi
pyrsistent 0.18.0 pypi_0 pypi
python 3.6.13 h12debd9_1
python-dateutil 2.9.0.post0 pypi_0 pypi
python_abi 3.6 2_cp36m conda-forge
pyyaml 6.0.1 pypi_0 pypi
pyzmq 25.1.2 pypi_0 pypi
qtconsole 5.2.2 pypi_0 pypi
qtpy 2.0.1 pypi_0 pypi
readline 8.2 h5eee18b_0
requests 2.27.1 pypi_0 pypi
requests-oauthlib 2.0.0 pypi_0 pypi
rsa 4.9 pypi_0 pypi
scikit-learn 0.24.2 pypi_0 pypi
scipy 1.4.1 pypi_0 pypi
send2trash 1.8.3 pypi_0 pypi
setuptools 58.0.4 py36h06a4308_0
six 1.16.0 pypi_0 pypi
sqlite 3.41.2 h5eee18b_0
tensorboard 2.2.2 pypi_0 pypi
tensorboard-plugin-wit 1.8.1 pypi_0 pypi
tensorflow 2.2.0 pypi_0 pypi
tensorflow-estimator 2.2.0 pypi_0 pypi
termcolor 1.1.0 pypi_0 pypi
terminado 0.12.1 pypi_0 pypi
testpath 0.6.0 pypi_0 pypi
threadpoolctl 3.1.0 pypi_0 pypi
tifffile 2020.9.3 pypi_0 pypi
tk 8.6.12 h1ccaba5_0
tornado 6.1 pypi_0 pypi
traitlets 4.3.3 pypi_0 pypi
typing-extensions 4.1.1 pypi_0 pypi
urllib3 1.26.18 pypi_0 pypi
wcwidth 0.2.13 pypi_0 pypi
webencodings 0.5.1 pypi_0 pypi
werkzeug 2.0.3 pypi_0 pypi
wheel 0.37.1 pyhd3eb1b0_0
widgetsnbextension 3.6.6 pypi_0 pypi
wrapt 1.16.0 pypi_0 pypi
xz 5.4.6 h5eee18b_0
zipp 3.6.0 pypi_0 pypi
zlib 1.2.13 hd590300_5 conda-forge
I then transformed the jupyter notebook into a .py file and just ran it from the CLI, and this is the Issues I am getting:
(IMC_Denoise2) [hd_bm327@o06c02(Singularity) Jupyter_Notebook_examples]$ python IMC_Denoise_Steombock.py
Using TensorFlow backend.
2024-04-10 10:10:01.130639: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcuda.so.1
2024-04-10 10:10:01.456994: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1561] Found device 0 with properties:
pciBusID: 0000:01:00.0 name: NVIDIA A40 computeCapability: 8.6
coreClock: 1.74GHz coreCount: 84 deviceMemorySize: 44.35GiB deviceMemoryBandwidth: 648.29GiB/s
2024-04-10 10:10:01.457149: W tensorflow/stream_executor/platform/default/dso_loader.cc:55] Could not load dynamic library 'libcudart.so.10.1'; dlerror: libcudart.so.10.1: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /.singularity.d/libs
2024-04-10 10:10:01.457200: W tensorflow/stream_executor/platform/default/dso_loader.cc:55] Could not load dynamic library 'libcublas.so.10'; dlerror: libcublas.so.10: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /.singularity.d/libs
2024-04-10 10:10:01.458161: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcufft.so.10
2024-04-10 10:10:01.458575: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcurand.so.10
2024-04-10 10:10:01.458637: W tensorflow/stream_executor/platform/default/dso_loader.cc:55] Could not load dynamic library 'libcusolver.so.10'; dlerror: libcusolver.so.10: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /.singularity.d/libs
2024-04-10 10:10:01.458687: W tensorflow/stream_executor/platform/default/dso_loader.cc:55] Could not load dynamic library 'libcusparse.so.10'; dlerror: libcusparse.so.10: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /.singularity.d/libs
2024-04-10 10:10:01.458732: W tensorflow/stream_executor/platform/default/dso_loader.cc:55] Could not load dynamic library 'libcudnn.so.7'; dlerror: libcudnn.so.7: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /.singularity.d/libs
2024-04-10 10:10:01.458738: W tensorflow/core/common_runtime/gpu/gpu_device.cc:1598] Cannot dlopen some GPU libraries. Please make sure the missing libraries mentioned above are installed properly if you would like to use GPU. Follow the guide at https://www.tensorflow.org/install/gpu for how to download and setup the required libraries for your platform.
Skipping registering GPU devices...
Image data loading completed!
The generated patches augmented.
The generated patches shuffled.
The shape of the generated training set for channel 6 is (10344, 64, 64).
2024-04-10 10:10:05.892646: I tensorflow/core/platform/cpu_feature_guard.cc:143] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2 FMA
2024-04-10 10:10:05.907950: I tensorflow/core/platform/profile_utils/cpu_utils.cc:102] CPU Frequency: 2600225000 Hz
2024-04-10 10:10:05.908304: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x1481b4000b60 initialized for platform Host (this does not guarantee that XLA will be used). Devices:
2024-04-10 10:10:05.908331: I tensorflow/compiler/xla/service/service.cc:176] StreamExecutor device (0): Host, Default Version
2024-04-10 10:10:05.909459: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1102] Device interconnect StreamExecutor with strength 1 edge matrix:
2024-04-10 10:10:05.909469: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1108]
The range value to the corresponding model is 86.11091079711915.
Input Channel Shape => (10344, 64, 64, 1)
Number of Training Examples: 8792
Number of Validation Examples: 1552
Each training patch with shape of (64, 64) will mask 8 pixels.
Training model...
Epoch 1/25
3/69 [>.............................] - ETA: 1:11 - loss: 0.6549^CTraceback (most recent call last):
It runs, but not on GPU. Our GPU is an NVIDIA A40.
I also tried to update cudNN and cudatoolkit to the latest versions, same error occurred.
Hi,
Thank you for your great work. I'm curious about the percentage of masked pixels in each patch—specifically, the rationale behind 0.2% (~8 pixels within a 64x64 patch). Any insights you can share on this?
Kind regards,
Rahil
Hi,
There are newer versions of tensorflow, cudnn and cudatoolkit compared to what IMC_Denoise uses. Could updating these packages in the script make the process of training go faster?
Hi,
Thanks very much for releasing this tool. It is proving to be very useful for our projects. For the majority of markers, the denoising performs nicely with the default parameters. However, in a channel where the majority of the field is true positive pixels, I see the the denoising actually erodes some of the real signal.
Denoised (Default parameters, color scale is same as above):
Do you have any suggestions for optimization of the denoising procedure for this particular issue?
Hey! Thank you for the great paper and package, I am keen to try this on my MIBI data. Unfortunately I get empty arrays when running the generate_patches_from_directory function. Images load okay but all values are changed to 0 at this step: patch_collect_sub = self.extract_patches_from_img(Img_DIMR, Row_range, Col_range)
The data is 1024x1024 and I get this output array([], shape=(0, 64, 64), dtype=float32)
Thank you so much for your help, I would appreciate any tips on running this type of data. Thanks!
Hi,
Great work on the development of DIMR and DeepSNiF for noise removal on IMC data. In the article, it says that the final layer of the U-Net uses softplus as it's activation function (log(1+exp(x))), which I believe this means that the denoised images will be log-transformed. On this basis, do I have to perform normalisation (percentile-method) on the pixel values prior inputting into DIMR and DeepSNiF? Or can I input raw images for DIMR and DeepSNiF, and from this, is it still suitable to perform log normalisation on the single cell data after segmentation?
These ideas might not make sense at all, as I am still relatively new to IMC data processing. Nevertheless, thanks for your help!
Hi,
I am interested in using the DIMR and DeepSNiF algorithms for my IMC data. I'd like to ask if the result is the same if I did training using a single marker or multi markers such as 30 or 40 markers?
Best regards
Currently, the number of steps per epoch is defined by
However, there is an edge case when the number of training samples is divisible by 128 (batch size), which results in WARNING:tensorflow:Your input ran out of data; interrupting training. Make sure that your dataset or generator can generate at least steps_per_epoch * epochs batches
, due to the floor and addition.
I wonder if it is necessary to include the last batch, which consists of ≤128 samples, and if very few samples in this batch could adversely affect loss for that last step. Then it would be appropriate to simply use floor. Otherwise, you can use ceiling to achieve the same logic without the edge case.
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.