Thanks to the author, this is a great article. I tried to reproduce it. However, the IU-Xray dataset I downloaded from the official website does not have folders. The 7470 images are all in one folder, but in reality the input needs to be divided into subfolders. I wonder how the author solved it at that time? Thank you very much!

Great work. Have you done any preprocessing on the mimic dataset?

The author of R2GenCMN mentioned that CE metrics are only applicable to MIMIC-CXR because the labeling schema of CheXpert is specifically designed for MIMIC-CXR, which is different from that of IU X-RAY.
So Will using CheXpert to label IU X-RAY directly introduce noise in practical applications?

Hi, thanks for your code! I have some questions about the model.
When we construct the prototype matrix(N_l x N_p x D), the 1xD vectors in it is derived from the whole image/sentence;
However, when conducting subsequent operations of the Cross-modal Prototype Querying and the Cross-modal Prototype Responding, it is to look for the most suitable vector in the prototype matrix for each patch or word. Does this sound not so matching? image -patch, sentence - word?

Dear author, I apologize for disturbing you, but I sincerely believe that this is a great piece of work.At present, I can run the project, and the results are quite satisfactory, but I would like to know how to insert an image to generate its corresponding report. Do I need to write another module myself? I look forward to your answer. Thank you very much.

Dear author, I am honored to read your excellent papers and codes. But I have some questions to ask you. I want to modify the generation method of .pt and .pickle files, but this part of code doesn't appear in the project, so could you please send me this part of code to help me refer to it?Thank you very much.
My email address is [email protected]

Dear Author，I'm really enthusiastic about your fantastic work，I just train IU-Xray dataset use your code，and I get the best result is
test_BLEU_1 : 0.432083288799186
test_BLEU_2 : 0.27494955972033686
test_BLEU_3 : 0.19862342676532663
test_BLEU_4 : 0.1530441598906721
test_METEOR : 0.17428250108359328
test_ROUGE_L : 0.3539938538730797
and in training process ，the terminal show ‘Validation performance didn't improve for 50 epochs. Training stops.‘
In the same time, I juest test the model with the trained models you provide, I get the result same as the paper,I‘m really confused
,So,could you supply more details about how to train the best model?
Sincerely

Dear author, I apologize for disturbing you, but I sincerely believe that this is a great piece of work. However, I am not getting the desired results on the iu_xray dataset, and I feel that it might be due to issues with the hyperparameters. Could you please guide me on what the ideal hyperparameters should be? Thank you very much.

Dear author, I hope this message finds you well. I have been reviewing your recent project that incorporates the IU-Xray dataset, and I am particularly intrigued by your data selection process. My query pertains to the specific parts of the dataset utilized in your analysis. Could you kindly confirm if your study exclusively employed the 'Findings' section from the reports within the IU-Xray dataset, while omitting the 'Impression' segment? I am eager to understand this aspect of your methodology as it holds significant implications for my own research endeavors. Your clarification would be immensely appreciated.

Hi, I have successfully reproduced your work and got the exact same results as described in your paper. But I found a phenomenon that when experimenting on the iu-xray dataset, the best model and results appeared in the 3rd epoch. Does this phenomenon indicate that the validity of the method proposed in the paper needs to be re-discussed? Can you explain to me whether this phenomenon is reasonable?
Generally speaking, using checkpoints obtained on the previous epochs to generate corresponding reports has poor diversity. I have tried using the best retrained model and the best model you provided to generate the corresponding report, and finally found that this is the case.
Below is an excerpt of the experiment log I got with the best results to demonstrate that I successfully reproduced the results.

07/24/2023 16:11:39 - INFO - modules.trainer - [3/30] Start to evaluate in the validation set.
07/24/2023 16:12:32 - INFO - modules.trainer - [3/30] Start to evaluate in the test set.
07/24/2023 16:13:57 - INFO - modules.trainer - epoch : 3
07/24/2023 16:13:57 - INFO - modules.trainer - ce_loss : 2.3583324741023457
07/24/2023 16:13:57 - INFO - modules.trainer - img_con : 0.010452255175282905
07/24/2023 16:13:57 - INFO - modules.trainer - txt_con : 0.02370573818510355
07/24/2023 16:13:57 - INFO - modules.trainer - img_bce_loss : 0.6931472420692444
07/24/2023 16:13:57 - INFO - modules.trainer - txt_bce_loss : 0.6931472420692444
07/24/2023 16:13:57 - INFO - modules.trainer - val_BLEU_1 : 0.4875411346726625
07/24/2023 16:13:57 - INFO - modules.trainer - val_BLEU_2 : 0.32324968962851985
07/24/2023 16:13:57 - INFO - modules.trainer - val_BLEU_3 : 0.2303989906968061
07/24/2023 16:13:57 - INFO - modules.trainer - val_BLEU_4 : 0.16892974375553144
07/24/2023 16:13:57 - INFO - modules.trainer - val_METEOR : 0.19912841341017073
07/24/2023 16:13:57 - INFO - modules.trainer - val_ROUGE_L : 0.3893886781595059
07/24/2023 16:13:57 - INFO - modules.trainer - test_BLEU_1 : 0.5247745358089907
07/24/2023 16:13:57 - INFO - modules.trainer - test_BLEU_2 : 0.35656897214407807
07/24/2023 16:13:57 - INFO - modules.trainer - test_BLEU_3 : 0.2620523629665125
07/24/2023 16:13:57 - INFO - modules.trainer - test_BLEU_4 : 0.19875032988045743
07/24/2023 16:13:57 - INFO - modules.trainer - test_METEOR : 0.21969653608856185
07/24/2023 16:13:57 - INFO - modules.trainer - test_ROUGE_L : 0.4113942119889325
07/24/2023 16:14:09 - INFO - modules.trainer - Saving checkpoint: /data/XProNet/results_RETRAIN_withReportGen/iu_xray/current_checkpoint.pth ...
07/24/2023 16:14:30 - INFO - modules.trainer - Saving current best: model_best.pth ...

Hello, I've attempted to reproduce the results seen in the report however after using your weights, I've been getting a shape error that is due to the size of the word mappings (token_to_id and id_to_token). I've made my own annotations.json file to split the data (70-10-20 as indicated in the paper) however the random split is causing a difference in how many words get embedded into meaningful tokens and how many are embedded as . May I ask how you split the data and if it's possible to have access to the original annotations.json file?

Hi ,Thank you for sharing your code. It's very nice work.
I'd like to ask you a few questions about CheXbert label. I think the label is obtained in the medical report. In the testing step, we didn't know it. But in the code, you use label ''output, _ = self.model(images, labels = labels, mode='sample')''.If I understand wrong, please let me know.
Thanks in avdance!

Dear author, I wonder if you got the prototype, did you first use a pretrained model to extract visual and textual features, then concatenate the visual and textual features, cluster 20 clusters with k-means? Normalize before clustering? Use each cluster center as a prototype? Could you please explain it in detail?Thank you very much.

Hi,

Thanks for sharing your work!

I already have access to the MIMIC data, but It is arranged differently.
Could you please upload the annotations.json file for that dataset? Unfortunately, there is no access via R2Gen/CMN.

Thank you

Hi, thanks for your code!
You note in Readme that the code references R2GenCMN.
However, there are some questions about the dataparallel code in R2GenCMN, as shown in #issue.

RuntimeError: Expected all tensors to be on the same device, but found at least two devices, cuda:1 and cuda:0!
(when checking argument for argument weight in method wrapper__cudnn_convolution)

And the code you provided is completely correct. Could you please tell me what modifications you have made?
Thank you in advance!

Hi, Jun:
Thanks for you release the code!!
I followed the sh file provided in git, but the results I reproduced on the IU_Xray dataset are very different from the results in the paper, did I miss something?