Design choice of trainable weights about medical-sam2 HOT 4 CLOSED

That's depends on your need. We find training 3D image encoder requires much more computational resource compared to 2D. Of course, fine-tune the entirely whole model is more likely for better performance (task specific). But considering the resource, we made such training decision.

from medical-sam2.

In this work, we train subparts except prompt encoder. Because we mention the terminology `one-prompt' proposed in the paper, so the occurrence of prompts should be sparse. That's why we froze prompt encoder. The image embedding's quality has an influence on memory quality, that's why we train both. For MSA, it attempts to efficiently adapt to the medical setting, so only train part of the layers in the image encoder. As medical image varies a lot to natural image (e.g. modality, ambiguous boundary etc.), we train the mask decoder in both MSA and MedSAM-2. In short, the scope and purpose of these two methods are different, so we make different decisions at training components. Also, you can freely freeze or defreeze any part depends on your need. Hope that helps!

from medical-sam2.

In this work, we train subparts except prompt encoder. Because we mention the terminology `one-prompt' proposed in the paper, so the occurrence of prompts should be sparse. That's why we froze prompt encoder. The image embedding's quality has an influence on memory quality, that's why we train both. For MSA, it attempts to efficiently adapt to the medical setting, so only train part of the layers in the image encoder. As medical image varies a lot to natural image (e.g. modality, ambiguous boundary etc.), we train the mask decoder in both MSA and MedSAM-2. In short, the scope and purpose of these two methods are different, so we make different decisions at training components. Also, you can freely freeze or defreeze any part depends on your need. Hope that helps!

Thank you for your reply
I apologize for the mistake in my question (updated).
I am trying to figure out: in this work, why the image encoder is trainable in 2D training but frozen in 3D training?

from medical-sam2.

Very clear.
Thank you so much.

from medical-sam2.