Code and data for multi-modal categorization of age-related macular degeneration (4 classes: normal, dry AMD, pcv, wet AMD)

MICAAI2019 paper | J-BHI 2022 article

• [2022-04-27] MMC-AMD has been accepted for publication as a regular paper in the IEEE Journal of Biomedical and Health Informatics (J-BHI)(IF: 5.772).

• Python-3.7.10
• CUDA-10.1
• Pytorch-1.1.0 & torchvision-0.3.0 conda install pytorch==1.1.0 torchvision==0.3.0 cudatoolkit=10.0
• Other Packages pip install -r requirements.txt

• MMC-AMD, a multi-modal fundus image set consisting of 1,093 color fundus photograph (CFP) images and 1,288 OCT B-scan images (~470MB). Freely available upon request and for research purposes only. Please submit your request via Google Form.
• MMC-AMD (splitA): An eye-based split of training / validation / test sets (zero eye overlap) Google drive
• MMC-AMD (splitAP): A patient-based split of training / validation / test sets (zero patient overlap) Google drive

By default, all the data folders are assumed to be placed (via symbolic links) at code/VisualSearch. The folders are organized as follows, see notebooks/count_data.ipynb for a code-level understanding.

./code/VisualSearch/
	mmc-amd/
		ImageData/
			cfp-clahe-448x448/
				f-*.jpg
			oct-median3x3-448x448/
				o-*.jpg
	mmc-amd-splitA-train/
		ImageSets/		(record image ID)
			cfp.txt
			oct.txt
		EyeSets/		(record eye ID)
			cfp.txt
			oct.txt
		SubjectSets/		(record patient ID)
			cfp.txt
			oct.txt
		annotations/		
			cfp.txt
			oct.txt
		ImageData		(symbolic link to $PATH/code/VisualSearch/mmc-amd/ImageData)
		
	mmc-amd-splitA-val/ (mmc-amd-splitA-test/)
		ImageSets/
			cfp.txt
			oct.txt
			mm.txt		(record cfp-oct pairs)
		EyeSets/
			cfp.txt
			oct.txt
		SubjectSets/
			cfp.txt
			oct.txt
		annotations/
			cfp.txt
			oct.txt
		ImageData		(symbolic link to $PATH/code/VisualSearch/mmc-amd/ImageData)

We provide a number of trained single-modal / multi-modal CNNs / CAMs-based GANs at Baidu pan (code: tkht), Google drive.

The test performance of these trained models on the two distinct data splits is as follows. Note that the numbers may differ (slightly) from that reported in the paper, wherein we report averaged result of three independent runs per model.

• notebooks/inference-and-eval-single-modal.ipynb: Run and evaluate a single-modal CNN
• notebooks/inference-and-eval-multi-modal.ipynb: Run and evaluate a multi-modal CNN
• notebooks/inference-and-synthesize-fake-images.ipynb: Run and synthesize fake images

• Make sure you have trained or downloaded CFP-CNN / OCT-CNN

bash scripts/do_generate_cam.sh cfp
bash scripts/do_generate_cam.sh oct

Once the CAMs are produced,

• Link the CAM data dir to code/camconditioned-pix2pixHD/datasets/$DATASET_NAME/train_A
• Link the image dir code/VisualSearch/mmc-amd/ImageData/$MODALITY to code/camconditioned-pix2pixHD/datasets/$DATASET_NAME/train_B

bash scripts/do_train_pix2pixHD.sh cfp
bash scripts/do_train_pix2pixHD.sh oct

bash scripts/do_img_synthesis.sh cfp
bash scripts/do_img_synthesis.sh oct

If you want to use the synthetic images in MM-CNN-da's training, please organize them as follows.

./code/VisualSearch/
	mmc-amd-splitA-syn/
		ImageSets/		(record image ID)
			cfp.txt
			oct.txt
		annotations/		
			cfp.txt
			oct.txt
		ImageData/
			cfp-clahe-448x448 		(symbolic link to the dir of synthetic cfp)
			oct-median3x3-448x448		(symbolic link to the dir of synthetic oct)
				

If you find this repository useful, please consider citing:

@inproceedings{miccai19-mmcamd,
  author    = {Weisen Wang and Zhiyan Xu and Weihong Yu and Jianchun Zhao and Jingyuan Yang and Feng He and Zhikun Yang and Di Chen and Dayong Ding and Youxin Chen and Xirong Li},
  title     = {Two-Stream {CNN} with Loose Pair Training for Multi-modal {AMD} Categorization},
  booktitle = {MICCAI},
  pages     = {156--164},
  doi = {10.1007/978-3-030-32239-7_18},
  year      = {2019},
}

@article{jbhi22-mmcamd,
  author={Weisen Wang and Xirong Li and Zhiyan Xu and Weihong Yu and Jianchun Zhao and Dayong Ding and Youxin Chen},
  journal={IEEE Journal of Biomedical and Health Informatics},
  title={Learning Two-Stream CNN for Multi-Modal Age-Related Macular Degeneration Categorization}, 
  year={2022},
  volume={26},
  number={8},
  pages={4111-4122},
  year={2022},
  doi={10.1109/JBHI.2022.3171523},
}

• The implementation of pix2pixHD was borrowed from pytorch-CycleGAN-and-pix2pix.
• This research was supported in part by the National Natural Science Foundation of China (No. 62172420, No. 61672523), Beijing Natural Science Foundation (No. 4202033), Beijing Natural Science Foundation Haidian Original Innovation Joint Fund (No. 19L2062), the Non-profit Central Research Institute Fund of Chinese Academy of Medical Sciences (No. 2018PT32029), CAMS Initiative for Innovative Medicine (CAMS-I2M, 2018-I2M-AI-001), and the Pharmaceutical Collaborative Innovation Research Project of Beijing Science and Technology Commission (No. Z191100007719002).