jeya-maria-jose / medical-transformer Goto Github PK

libpng warning: iCCP: known incorrect sRGB profile
epoch [0/400], loss:0.2294
libpng warning: iCCP: known incorrect sRGB profile
libpng warning: iCCP: known incorrect sRGB profile
epoch [1/400], loss:0.0000
libpng warning: iCCP: known incorrect sRGB profile
epoch [2/400], loss:0.0000
libpng warning: iCCP: known incorrect sRGB profile
epoch [3/400], loss:0.0000
libpng warning: iCCP: known incorrect sRGB profile
epoch [4/400], loss:0.0000
libpng warning: iCCP: known incorrect sRGB profile
epoch [5/400], loss:0.0000
libpng warning: iCCP: known incorrect sRGB profile
epoch [6/400], loss:0.0000
libpng warning: iCCP: known incorrect sRGB profile
epoch [7/400], loss:0.0000
libpng warning: iCCP: known incorrect sRGB profile
epoch [8/400], loss:0.0000
libpng warning: iCCP: known incorrect sRGB profile
epoch [9/400], loss:0.0000
libpng warning: iCCP: known incorrect sRGB profile
epoch [10/400], loss:0.0000
libpng warning: iCCP: known incorrect sRGB profile
libpng warning: iCCP: known incorrect sRGB profile
epoch [11/400], loss:0.0000
libpng warning: iCCP: known incorrect sRGB profile
epoch [12/400], loss:0.0000

dear all,
I use GTX 2080Ti to train.
I had the problem below: imgsize = 512
How to solve that?
(pytorch_unet) D:\6449 EBC\segmentation\Medical-Transformer-main>python train.py --train_dataset "data_6449_big/train" --val_dataset "data_6449_big/testB" --direc 'result' --batch_size 1 --epoch 200 --save_freq 10 --modelname "gatedaxialunet" --learning_rate 0.001 --imgsize 512 --gray "no"
Total_params: 1360642
Traceback (most recent call last):
File "train.py", line 140, in
output = model(X_batch)
File "C:\Users\USER\anaconda3\envs\pytorch_unet\lib\site-packages\torch\nn\modules\module.py", line 1051, in _call_impl
return forward_call(*input, **kwargs)
File "D:\6449 EBC\segmentation\Medical-Transformer-main\lib\models\axialnet.py", line 507, in forward
return self._forward_impl(x)
File "D:\6449 EBC\segmentation\Medical-Transformer-main\lib\models\axialnet.py", line 487, in _forward_impl
x2 = self.layer2(x1)
File "C:\Users\USER\anaconda3\envs\pytorch_unet\lib\site-packages\torch\nn\modules\module.py", line 1051, in _call_impl
return forward_call(*input, **kwargs)
File "C:\Users\USER\anaconda3\envs\pytorch_unet\lib\site-packages\torch\nn\modules\container.py", line 139, in forward
input = module(input)
File "C:\Users\USER\anaconda3\envs\pytorch_unet\lib\site-packages\torch\nn\modules\module.py", line 1051, in _call_impl
return forward_call(*input, **kwargs)
File "D:\6449 EBC\segmentation\Medical-Transformer-main\lib\models\axialnet.py", line 331, in forward
out = self.hight_block(out)
File "C:\Users\USER\anaconda3\envs\pytorch_unet\lib\site-packages\torch\nn\modules\module.py", line 1051, in _call_impl
return forward_call(*input, **kwargs)
File "D:\6449 EBC\segmentation\Medical-Transformer-main\lib\models\axialnet.py", line 166, in forward
stacked_similarity = torch.cat([qk, qr, kr], dim=1)
RuntimeError: CUDA out of memory. Tried to allocate 1.50 GiB (GPU 0; 11.00 GiB total capacity; 6.74 GiB already allocated; 777.94 MiB free; 8.13 GiB reserved in total by PyTorch)

Thanks for your great work, and I need some help on training Glas datset. Could you please tell me the image size you used for training Glas dataset?Cause the images in Glas dataset tend to have different size, 775 x 522, 589 x 453 and 567 x 430, which need to be converted to the same shape before sending to the network, and 128x128 is just to small for training Glas dataset,. Thanks.

I could not see the label in the Annotations folder. In this folder has only .xml file. Please help me have labels in order train. Thank you.

Thank you.

Is some bugs in train.py?
For example——parser.add_argument('--train_dataset', required=True, type=str)
I think here should exist the path of dataset,and when i add the path,there occurs other errors, that i shows as belows:
usage: train.py [-h] [-j N] [--epochs N] [--start-epoch N] [-b N]
[--learning_rate LR] [--momentum M] [--weight-decay W]
--train_dataset TRAIN_DATASET [--val_dataset VAL_DATASET]
[--save_freq SAVE_FREQ] [--modelname MODELNAME] [--cuda CUDA]
[--aug AUG] [--load LOAD] [--save SAVE] [--direc DIREC]
[--crop CROP] [--imgsize IMGSIZE] [--device DEVICE]
[--gray GRAY]
train.py: error: the following arguments are required: --train_dataset
An exception has occurred, use %tb to see the full traceback.
i have tried my best to try to understand utils.py(class ImageToImage2D(Dataset)),and cannot find the reason of debugs
Could author or friends give me a hand? Thank you very much...

Without GPU, can only CPU run，What should I do?

Thank you for your excellent code. Could you tell me what is the image size when you used to train your network. If I want to use my own image, for example the image with size 512 x 512, what parameters should I modify? Since when I try to re-implement your project, I encountered some dimensions issues, like

RuntimeError: einsum() operands do not broadcast with remapped shapes [original->remapped]: [1024, 8, 1, 512]->[1024, 8, 512, 1, 1] [1, 128, 128]->[1, 1, 128, 128, 1]

Could you give me some hints?

Traceback (most recent call last):
File "train.py", line 140, in
output = model(X_batch)
File "/home/zzp/.local/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1051, in _call_impl
return forward_call(*input, **kwargs)
File "/home/zzp/2t/binzhang/Medical-Transformer/lib/models/axialnet.py", line 507, in forward
return self._forward_impl(x)
File "/home/zzp/2t/binzhang/Medical-Transformer/lib/models/axialnet.py", line 485, in _forward_impl
x1 = self.layer1(x)
File "/home/zzp/.local/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1051, in _call_impl
return forward_call(*input, **kwargs)
File "/home/zzp/.local/lib/python3.8/site-packages/torch/nn/modules/container.py", line 139, in forward
input = module(input)
File "/home/zzp/.local/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1051, in _call_impl
return forward_call(*input, **kwargs)
File "/home/zzp/2t/binzhang/Medical-Transformer/lib/models/axialnet.py", line 331, in forward
out = self.hight_block(out)
File "/home/zzp/.local/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1051, in _call_impl
return forward_call(*input, **kwargs)
File "/home/zzp/2t/binzhang/Medical-Transformer/lib/models/axialnet.py", line 157, in forward
qr = torch.einsum('bgci,cij->bgij', q, q_embedding)
File "/home/zzp/.local/lib/python3.8/site-packages/torch/functional.py", line 299, in einsum
return _VF.einsum(equation, operands) # type: ignore[attr-defined]
RuntimeError: einsum(): operands do not broadcast with remapped shapes [original->remapped]: [2000, 8, 1, 500]->[2000, 8, 500, 1, 1] [1, 64, 64]->[1, 1, 64, 64, 1]

Hi,
First of all, congrats on your work on Medical Transformer, and thanks for sharing the code. Currently, I am working on transformer-based segmentation tasks and I would like to use your code on my data. Is it possible to use this code for multi-class segmentation?

When the ‘modelname’ is set to "logo", should the size of the parameter ‘crop’ be set to a quarter of ‘imgsize’? When I set ‘imgsize’ to 512 and ‘crop’ to 128 like this, the following error occurred:
RuntimeError: einsum() operands do not broadcast with remapped shapes [original->remapped]: [64, 8, 1, 64]->[64, 8, 64, 1, 1] [1, 256, 256]->[1, 1, 256, 256, 1]
looking forward to your reply!

Hi, sorry to bother you, really appreciate your work. I recently trained the axialunet with input size 473, and it gives me a cuda out of memory error, I also noticed that you seems to comment out the decoder1, decoder2, decoder3 code in medt_net, could you please tell me why? Is it because the memory consumption problem?Thanks in advance.

This is my testing command: python test.py --loaddirec "./medtfinal_model.pth" --val_dataset "./val/" --batch_size 1 --modelname "gatedaxialunet" --imgsize 128 --gray "no"

But what makes me strange is that it seems to have some problems, can you help me solve it?

Total_params: 1560116
File "E:/宋应清/Medical-Transformer-main/train.py", line 130, in
for batch_idx, (X_batch, y_batch, *rest) in enumerate(dataloader):
File "D:\Users\DELL\Anaconda3\envs\syq\lib\site-packages\torch\utils\data\dataloader.py", line 517, in next
data = self._next_data()
File "D:\Users\DELL\Anaconda3\envs\syq\lib\site-packages\torch\utils\data\dataloader.py", line 557, in _next_data
data = self._dataset_fetcher.fetch(index) # may raise StopIteration
File "D:\Users\DELL\Anaconda3\envs\syq\lib\site-packages\torch\utils\data_utils\fetch.py", line 44, in fetch
data = [self.dataset[idx] for idx in possibly_batched_index]
File "D:\Users\DELL\Anaconda3\envs\syq\lib\site-packages\torch\utils\data_utils\fetch.py", line 44, in
data = [self.dataset[idx] for idx in possibly_batched_index]
File "E:\宋应清\Medical-Transformer-main\utils.py", line 156, in getitem
mask[mask<=127] = 0
TypeError: '<=' not supported between instances of 'NoneType' and 'int'

hello, i use GLAS resize dataset, but whith i run train.py, the loss is nan.

my env is windows 10, and:

dear @jeya-maria-jose,
Can you help me,
I had problem about test.py below:
(medt) D:\segmentation\Medical-Transformer-main>python test.py --loaddirec "resultfinal_model.pth" --val_dataset "data_6449_s256/test" --direc '290/gatedaxialunet.pth' --batch_size 1 --modelname "gatedaxialunet" --imgsize 256 --gray "no"
None
Traceback (most recent call last):
File "test.py", line 82, in
train_dataset = ImageToImage2D(args.train_dataset, tf_train)
File "D:\segmentation\Medical-Transformer-main\utils.py", line 131, in init
self.input_path = os.path.join(dataset_path, 'img')
File "C:\Users\DELL\anaconda3\envs\medt\lib\ntpath.py", line 76, in join
path = os.fspath(path)
TypeError: expected str, bytes or os.PathLike object, not NoneType

Hi, Jose. This work attracts me a lot and I attempt to test it on another dataset. While revising the source code, I found the gated axial attention layer seems to be only used in the global branch (link). The local branch keeps using the axial attention module (link). Because this may be not referred in the paper, I just want to check whether I ignore something here. Thanks in advance.

how could I solve this problm?

RuntimeError: size of dimension does not match previous size, operand 1, dim 1

Total_params: 1326850
Traceback (most recent call last):
File "train.py", line 130, in
for batch_idx, (X_batch, y_batch, *rest) in enumerate(dataloader):
File "D:\anaconda1\envs\Medical-transformer\lib\site-packages\torch\utils\data\dataloader.py", line 521, in next
data = self._next_data()
File "D:\anaconda1\envs\Medical-transformer\lib\site-packages\torch\utils\data\dataloader.py", line 561, in _next_data
data = self._dataset_fetcher.fetch(index) # may raise StopIteration
File "D:\anaconda1\envs\Medical-transformer\lib\site-packages\torch\utils\data_utils\fetch.py", line 44, in fetch
data = [self.dataset[idx] for idx in possibly_batched_index]
File "D:\anaconda1\envs\Medical-transformer\lib\site-packages\torch\utils\data_utils\fetch.py", line 44, in
data = [self.dataset[idx] for idx in possibly_batched_index]
File "E:\Medical-Transformer-main\utils.py", line 156, in getitem
mask[mask<=127] = 0
TypeError: '<=' not supported between instances of 'NoneType' and 'int'

How does the author solve this error report? Is it a problem of data set processing? How to solve it？

Excuse me, for the gated axial-attention formula, I can understant the setting of the gating parameters in the three positions of Gqr, Gkr and Gr, but why do you add the gating parameters on v, which has no position information.
Thank you!

hello,i want to know what mean of --gray parameter, --gray yes is using 8 bit image training?

Hi, I have a ground-truth with 3 classes including background with values 0,127,255. As mentioned in #43 I changed num_classes=3 in axialnet.py
In utils.py,

which makes the ground-truth with values 0 and 1 but I should have 0,1 and 2 for my case(with 3 classes).

I tried doing this
mask[mask<127] = 0
mask[mask==127] = 1
mask[mask>127] = 2

But I got this error. Could you please help me with this?

Hello Jose, thank you very much for your codes. I have some questions here. In your paper, the local branch has 5 blocks of encoder and 5 blocks of decoder. But in your code, the local branch seems only to have 4 blocks of encoder like:
" img_size_p = img_size // 4
self.layer1_p = self._make_layer(block_2, int(128 * s), layers[0], kernel_size= (img_size_p//2))
self.layer2_p = self._make_layer(block_2, int(256 * s), layers[1], stride=2, kernel_size=(img_size_p//2),
dilate=replace_stride_with_dilation[0])
self.layer3_p = self._make_layer(block_2, int(512 * s), layers[2], stride=2, kernel_size=(img_size_p//4),
dilate=replace_stride_with_dilation[1])
self.layer4_p = self._make_layer(block_2, int(1024 * s), layers[3], stride=2, kernel_size=(img_size_p//8),
dilate=replace_stride_with_dilation[2]) ". Are the codes you provided just to illustrate the feasibility of MedT? In fact, the codes of this part are not consistent with the actual structure of MedT in your paper?

Why the block applied in the global branch is AxialBlock_dynamic while AxialBlock_wopos is used in the local branch?

Hello, I run the training and testing on the MoNuSeg dataset using the command you provided (set modelname as MedT).

The testing ground truth is from 0-255, so I used the following to convert it:

gt_arr = np.where(gt_arr>127, 1, 0)

Then I compute the f1 score of the val images, which is less than 0.6. Did I do anything wrong?

Hello author. Thanks for your code. Do you just calculate the attention on the patch image whose size is 32x32 and attention between patches are not considerded? Look at the following codes.
"for i in range(0,4):
for j in range(0,4):
x_p = xin[:,:,32i:32(i+1),32j:32(j+1)] #4x1x32x32
# begin patch wise
x_p = self.conv1_p(x_p)
x_p = self.bn1_p(x_p)
# x = F.max_pool2d(x,2,2)
x_p = self.relu(x_p) #4x64x16x16

            x_p = self.conv2_p(x_p) #4x128x16x16
            x_p = self.bn2_p(x_p)
            # x = F.max_pool2d(x,2,2)
            x_p = self.relu(x_p)
            x_p = self.conv3_p(x_p)#4x64x16x16
            x_p = self.bn3_p(x_p)
            # x = F.max_pool2d(x,2,2)
            x_p = self.relu(x_p)
            
            # x = self.maxpool(x)
            # pdb.set_trace()
            x1_p = self.layer1_p(x_p) #1 #4x32x16x16
            # print(x1.shape)
            x2_p = self.layer2_p(x1_p) #2 #4x64x8x8
            # print(x2.shape)
            x3_p = self.layer3_p(x2_p) #4 #4x128x4x4
            # # print(x3.shape)
            x4_p = self.layer4_p(x3_p) #1  #4x256x2x2
            
            x_p = F.relu(F.interpolate(self.decoder1_p(x4_p), scale_factor=(2,2), mode ='bilinear'))#4x256x2x2
            x_p = torch.add(x_p, x4_p)
            x_p = F.relu(F.interpolate(self.decoder2_p(x_p) , scale_factor=(2,2), mode ='bilinear'))#4x128x4x4
            x_p = torch.add(x_p, x3_p)
            x_p = F.relu(F.interpolate(self.decoder3_p(x_p) , scale_factor=(2,2), mode ='bilinear'))#4x64x8x8
            x_p = torch.add(x_p, x2_p)
            x_p = F.relu(F.interpolate(self.decoder4_p(x_p) , scale_factor=(2,2), mode ='bilinear'))#4x32x16x16
            x_p = torch.add(x_p, x1_p)
            x_p = F.relu(F.interpolate(self.decoder5_p(x_p) , scale_factor=(2,2), mode ='bilinear'))#4x16x32x32
            
            x_loc[:,:,32*i:32*(i+1),32*j:32*(j+1)] = x_p"

I have an unkind request.Can you provide train .py files about unet++ etc.

On line 87 of train.py and 82 of test.py

tf_train = JointTransform2D(crop=crop, p_flip=0.5, color_jitter_params=None, long_mask=True)
tf_val = JointTransform2D(crop=crop, p_flip=0, color_jitter_params=None, long_mask=True)
train_dataset = ImageToImage2D(args.train_dataset, tf_val)  # Here
val_dataset = ImageToImage2D(args.val_dataset, tf_val)

FileNotFoundError Traceback (most recent call last)
in ()
4 I = np.array(Image.open('/content/test folder/labelcol/'+i))
5 I=np.where(I>=127,1,0)
----> 6 L = np.array(Image.open('/content/test_results/'+i))
7 L=np.where(L>=127,1,0)
8 y_true,y_pred = I.flatten(),L.flatten()

/usr/local/lib/python3.7/dist-packages/PIL/Image.py in open(fp, mode)
2841
2842 if filename:
-> 2843 fp = builtins.open(filename, "rb")
2844 exclusive_fp = True
2845

FileNotFoundError: [Errno 2] No such file or directory: '/content/test_results/0004.png'

code of class qkv_transform(nn.Conv1d) is missing from Medical-Transformer/lib/models/utils.py file.

Hello, author. Thank you for your code. CUDA has been used and the batch size has been adjusted to 4, but the training speed of the model is still very slow. Are you in this situation? How can I solve this problem? Look forward to your reply.

Hi, dear authors, I am following your excellent work. But I find no information about how to split the train set and test set, which is the key for a fair comparison of different models. So would you please add some instructions?

How is the author's data processed? I downloaded the GLAS dataset, but I kept reporting errors when running
Can the author provide a processed dataset?
Total_params: 1326850
Traceback (most recent call last):
File "train.py", line 130, in
for batch_idx, (X_batch, y_batch, *rest) in enumerate(dataloader):
File "D:\anaconda1\envs\Medical-transformer\lib\site-packages\torch\utils\data\dataloader.py", line 521, in next
data = self._next_data()
File "D:\anaconda1\envs\Medical-transformer\lib\site-packages\torch\utils\data\dataloader.py", line 561, in _next_data
data = self._dataset_fetcher.fetch(index) # may raise StopIteration
File "D:\anaconda1\envs\Medical-transformer\lib\site-packages\torch\utils\data_utils\fetch.py", line 44, in fetch
data = [self.dataset[idx] for idx in possibly_batched_index]
File "D:\anaconda1\envs\Medical-transformer\lib\site-packages\torch\utils\data_utils\fetch.py", line 44, in
data = [self.dataset[idx] for idx in possibly_batched_index]
File "E:\Medical-Transformer-main\utils.py", line 156, in getitem
mask[mask<=127] = 0
TypeError: '<=' not supported between instances of 'NoneType' and 'int'

Brain US 数据集啥时候开啊

Thanks for your nice work . Since I just got in touch with this research direction, I meet some problems on training Glas dataset. Hope for your help!
I don’t know what went wrong, which made it impossible to train normally.

my env is Ubuntu18.04, All dependencies are consistent with your suggestions.

Glas dataset has been prepared in the following format

hello, I have a question, the qkv_transformer function is empty, what should I do next？

Hello, I trained U-Net under the same conditions, and its results are much higher than MedT and the score of U-Net which proposed in your paper. So can did you train the U-Net?

Hi, sorry if this is an obvious question, but why do you delay training the gates until the 10th epoch? Thanks, and great work!

Hi,
I have downloaded the GLAS Dataset, but I find the label has 10 classes, which are 0, 1, 2, 3, ..., 9. So, which class should be saved for a binary segmentation? Thanks.

Hello, thanks for your work!
The IoU of MoNuSeg Dataset in your colab code is 0.8400399344308036, while the IoU is 66.17% in your paper. Could you please explain what causes this difference?

thank you for your work, i have a question. Can i use pixel like 1024*988,1033*333 etc.

Thank you for your excellent contribution, and I have reproduced the experiment of the thesis. An error occurred during the test. The test result is quite different from the test result of the thesis. Will you update test.py recently，please.

I have a question about a code for masks in ImageToImage2D function, which are as follows:

class ImageToImage2D(Dataset):
...
mask[mask<127] = 0
mask[mask>=127] = 1
...

Sorry, I don't understand why to do this process to the mask. I will be very thanked for it if you can help me to work out.

Thanks for this amazing work. Would it be possible for you to share the link to the trained model. I wanted to see the inference results. Thanks in advance

I run the test.py on validation dataset of MoNuSegData, which generate some prediction images. I used these prediction image and ground truth of validation dataset to calculate IoU, but i achieved the IoU that is not same as yours. I think that i have some misunderstanding about the way of calculating IoU, so i want to ask how to calculate IoU.

Hi,Thanks for your wonderful work, I have a question as followed:Your work is for 2D Medical image segmentation or 3D Medical image segmentation?

While using the colab version of the code, the code works fine up to training. But while executing the following shell:
command2="python Medical-Transformer/test.py \ --loaddirec \"{}\" \ --val_dataset \"{}\" \ --direc '{}' \ --batch_size 1 \ --modelname \"MedT\" \ --imgsize 128 \ --gray \"no\"".format('/content/train results/390/MedT.pth',test_DIR,test_rese_DIR) !{command2}

the following error occurs:

Traceback (most recent call last):
File "Medical-Transformer/test.py", line 82, in
train_dataset = ImageToImage2D(args.train_dataset, tf_val)
File "/content/Medical-Transformer/utils.py", line 130, in init
self.input_path = os.path.join(dataset_path, '/img')
File "/usr/lib/python3.7/posixpath.py", line 80, in join
a = os.fspath(a)
TypeError: expected str, bytes or os.PathLike object, not NoneType

Please help :)

pytorch == 1.8.0, torchvision == 0.9.0
Is it caused by the environment ？