puzzles about this fancy paper about tandemnet HOT 4 CLOSED

1. The text attention is meaningful when text input is given.
2. For most experiments, training is done given by both image and text. w/o text and w/ text means the situation in the testing stage.
3. No model difference, just do not send the text input (by setting the text feature matrix to zero).

from tandemnet.

@zizhaozhang
I am very interested in this paper. Hope you can provide some answers for my questions, thanks a lot!

1. so one of the idea this paper wants to tell us is that , using both image and text to train a model, and the image classification accuracy will increase? What reason can explain this accuracy increase?

2. it seems that your report can be summarized as 5 2-class or 3-class classification problems. So in the real medical report, I wonder that, does the doctor write sentences like "moderate crowding of the nuclei can be seen" , or write a crowding label(for example, 1 indicates mild crowding, 2 indicates moderate crowding, 3 indicates severe crowding)? If write labels, why not regard this a pure image classification problem?( I mean 5 classification problems , regarding the 5 cell appearances)

3. Besides, I observed that the descriptions are mainly about the cell appearance features, so why not directly using cell segmentation results to get the cell appearance features, since this seems more reasonable and reliable than learning from the report? (Because the cell segmentation results can not be integrated into a deep learning framework?)

from tandemnet.

Thanks !

1. The introduced text knowledge
2. Please note that we are not trying to solve the problem of this dataset. Our model is designed to input free text. For pathology images, it is true for pathologist to write this kind of report. They prefer to not change too much. You can find evidence from a pathology book.
3. It is a fundamental logic question. Do you think image recognition is useful? Why we just do object detection?
   Pls ask further paper question using email. This is for code issue :)

from tandemnet.

@zizhaozhang
I have already emailed you([email protected]) for further questions. Waiting for your answer. Thanks!

from tandemnet.