We developed a Multi-modal Response Prediction (MRP) system for instant prediction in breast cancer treatment with Neoadjuvant Therapy (NAT). Using longitudinal multidisciplinary data from 3,352 breast cancer patients, MRP mimics physician assessments, aided by a cross-modal learning module for facilitating the MRP practicality in multi-center hospitals. Validated across centers and reader studies, MRP exhibited comparable robustness and generalizability to breast radiologists, significantly outperforming humans in pCR prediction in the Pre-NAT phase. Clinical utility assessment shows MRP's potential: reducing treatment toxicity by 35.8% in Pre-NAT and preventing surgeries by 16.7% in Post-NAT, without mispredictions. Our work enhances AI applications in personalized treatment decisions.

Start by installing PyTorch 1.8.1 with the right CUDA version, then clone this repository and install the dependencies.

$ conda install pytorch==1.8.1 torchvision==0.9.1 torchaudio==0.8.1 cudatoolkit=11.1 -c pytorch
$ pip install [email protected]:yawwG/MRP.git
$ conda env create -f environment.yml

This codebase has been developed with python version 3.7, PyTorch version 1.8.1, CUDA 11.1 and pytorch-lightning 1.5.9. Example configurations for MG-based and MRI-based classification can be found in the ./configs. All training and testing are done using the run.py script. For more documentation, please run:

python run.py --help

The preprocessing steps for each dataset (MRI and Mammogram) can be found in datasets/image_dataset.py The dataset using is specified in config.yaml by key("dataset").

Training the MRI and Mammogram based model for pCR prediction with the following command:

python run.py -c imrrhpc.yaml --train --test
python run.py -c imgrhpc.yaml --train --test

To ensure that the model's interpretability and predictable performance, we explicitly demonstrate the contribution of multi-modalities in the model's training.

python ./utils/contribution.py

We explored in two specific clinical scenarios: personalizing Pre-/Mid-NAT management of non-pCR patients to avoid toxic therapy and optimizing Post-NAT management of pCR patients to reduce unnecessary surgeries.

python ./utils/dca_analysis.py

If you have any questions please contact us.

Email: [email protected] (Ritse Mann); [email protected] (Tao Tan); [email protected] (Yuan Gao)

Links: Netherlands Cancer Institute, Radboud University Medical Center, Maastricht University, St Joseph’s Healthcare Hamilton and The University of Hong Kong