Promblem with train.py about whole-slide-cnn HOT 9 CLOSED

Thank you very much Chi-Chung. Every thing is fine now

from whole-slide-cnn.

Hi TargosLi,

Thank you for trying our codes.

From our experience, SIGKILL usually occurs when the program has run out of host memory. You can confirm this by observing the memory usage in real-time using htop or top.

If that is the case, you can try reducing INPUT_SIZE and RESIZE_RATIO in your training config to lower the memory consumption.

Best,
Chi-Chung

from whole-slide-cnn.

Thank you Chi-Chung,
I have reduced INPUT_SIZE and RESIZE_RATIO. It works.
but now I have the following problems:

1. W tensorflow/stream_executor/gpu/asm_compiler.cc:235] Your CUDA software stack is old. We fallback to the NVIDIA driver for some compilation.Update your CUDA version to get the best performance. The ptxas error was: ptxas fatal : Value 'sm_86' is not defined for option 'gpu-name'
2. W tensorflow/core/framework/op_kernel.cc:1751] Invalid argument: ValueError: need at least one array to concatenate
   Traceback (most recent call last):
   File "......dataloader.py", line 202, in getitem
   mil_infer_res = np.concatenate(mil_infer_res, axis=0)

File "<array_function internals>", line 5, in concatenate

ValueError: need at least one array to concatenate

Thank you

from whole-slide-cnn.

Hi TargosLi,

The error occurs when the program cannot find any foreground patch (patch with at least one RGB value larger than 220) for MIL training.
Please make sure your input slide is not purely empty.
Otherwise, you could try raising MIL_WHITE_THRESHOLD in whole_slide_cnn/dataloader.py:126 to a higher value to prevent background filtering.

Best,
Chi-Chung

from whole-slide-cnn.

Hi Chi-Chung,
thank you for your quick antwort.
I have raised MIL_WHITE_THRESHOLD in whole_slide_cnn/dataloader.py:126 to 300
another error:
Test Throughput: 74.15332570876468 (-53.96235541950328 - 202.26900683703263) images/minutes.
1/1 [==============================] - 630s 630s/step - loss: 0.3022 - accuracy: 1.0000 - val_loss: 0.9259 - val_accuracy: 0.4246

Epoch 00001: val_loss improved from inf to 0.92589, saving model to result_wholeslide_Targos/model.h5/PycharmProjects/whole-slide-cnn-main/venv/lib/python3.8/site-packages/numpy/core/fromnumeric.py:3372: RuntimeWarning: Mean of empty slice.
return _methods._mean(a, axis=axis, dtype=dtype,
/home/de1119151/PycharmProjects/whole-slide-cnn-main/venv/lib/python3.8/site-packages/numpy/core/_methods.py:170: RuntimeWarning: invalid value encountered in double_scalars
ret = ret.dtype.type(ret / rcount)
/home/de1119151/PycharmProjects/whole-slide-cnn-main/venv/lib/python3.8/site-packages/numpy/core/_methods.py:233: RuntimeWarning: Degrees of freedom <= 0 for slice
ret = _var(a, axis=axis, dtype=dtype, out=out, ddof=ddof,
/home/de1119151/PycharmProjects/whole-slide-cnn-main/venv/lib/python3.8/site-packages/numpy/core/_methods.py:194: RuntimeWarning: invalid value encountered in true_divide
arrmean = um.true_divide(
/home/de1119151/PycharmProjects/whole-slide-cnn-main/venv/lib/python3.8/site-packages/numpy/core/_methods.py:226: RuntimeWarning: invalid value encountered in double_scalars
ret = ret.dtype.type(ret / rcount)
Train Throughput: nan (nan - nan) images/minutes.

Process finished with exit code 0

from whole-slide-cnn.

Hi TargoLi,

The log you provided seems normal to me.
The warning messages are all benign, and the exit code usually indicates the program stops normally.
Do you set NUM_EPOCH = 1 so that the training process ends after one epoch?

Best,
Chi-Chung

from whole-slide-cnn.

Hello,
now I set USE_HMS true. other parameters are not changed. what should I change to avoid the error below?
here is error:
Traceback (most recent call last):
File "...whole_slide_cnn/train.py", line 128, in
model = build_model(
File "...whole_slide_cnn/model.py", line 129, in build_model
conv_block = get_conv_block(input_shape)
File "..../whole_slide_cnn/model.py", line 85, in get_conv_block
conv_block = model_fn(
File ".../whole_slide_cnn/model.py", line 26, in
"fixup_resnet50": lambda *args, **kwargs: ResNet50(
File "...whole_slide_cnn/resnet.py", line 557, in ResNet50
return ResNet(stack_fn, False, True, 'resnet50',
File "...whole_slide_cnn/resnet.py", line 436, in ResNet
x = _ZeroPadding2D(padding=((3, 3), (3, 3)), name='conv1_pad')(x)
File "...whole_slide_cnn/huge_layer_wrapper.py", line 206, in call
res = super(HugeLayerWrapper, self).call(inputs, **kwargs)
File "...venv/lib/python3.8/site-packages/tensorflow/python/keras/engine/base_layer.py", line 1012, in call
outputs = call_fn(inputs, *args, **kwargs)
File ".../whole_slide_cnn/huge_layer_wrapper.py", line 267, in call
output_tensor_list = self._do_padding(inputs, **kwargs)
File "...whole_slide_cnn/huge_layer_wrapper.py", line 517, in _do_padding
self.layer.compute_output_shape(self._get_shape(inputs)),
File "/...venv/lib/python3.8/site-packages/tensorflow/python/keras/layers/convolutional.py", line 2868, in compute_output_shape
if input_shape[1] is not None:
IndexError: list index out of range

from whole-slide-cnn.

Hi @TargosLi, would you share your training config and the Tensorflow version you use?

Chi-Chung

from whole-slide-cnn.

Thank you chenchc,
sorry I was on vacation, so I didn't reply your question.
Tensorflow version is 2.4.1
training config
config.txt
I tried this config, and
Traceback (most recent call last):
File "/home/de1119151/PycharmProjects/whole-slide-cnn-main/whole_slide_cnn/train.py", line 128, in
model = build_model(
File "/home/de1119151/PycharmProjects/whole-slide-cnn-main/whole_slide_cnn/model.py", line 129, in build_model
conv_block = get_conv_block(input_shape)
File "/home/de1119151/PycharmProjects/whole-slide-cnn-main/whole_slide_cnn/model.py", line 85, in get_conv_block
conv_block = model_fn(
File "/home/de1119151/PycharmProjects/whole-slide-cnn-main/whole_slide_cnn/model.py", line 26, in
"fixup_resnet50": lambda *args, **kwargs: ResNet50(
File "/home/de1119151/PycharmProjects/whole-slide-cnn-main/whole_slide_cnn/resnet.py", line 557, in ResNet50
return ResNet(stack_fn, False, True, 'resnet50',
File "/home/de1119151/PycharmProjects/whole-slide-cnn-main/whole_slide_cnn/resnet.py", line 436, in ResNet
x = _ZeroPadding2D(padding=((3, 3), (3, 3)), name='conv1_pad')(x)
File "/home/de1119151/PycharmProjects/whole-slide-cnn-main/whole_slide_cnn/huge_layer_wrapper.py", line 206, in call
res = super(HugeLayerWrapper, self).call(inputs, **kwargs)
File "/home/de1119151/PycharmProjects/whole-slide-cnn-main/venv/lib/python3.8/site-packages/tensorflow/python/keras/engine/base_layer.py", line 1012, in call
outputs = call_fn(inputs, *args, **kwargs)
File "/home/de1119151/PycharmProjects/whole-slide-cnn-main/whole_slide_cnn/huge_layer_wrapper.py", line 267, in call
output_tensor_list = self._do_padding(inputs, **kwargs)
File "/home/de1119151/PycharmProjects/whole-slide-cnn-main/whole_slide_cnn/huge_layer_wrapper.py", line 517, in _do_padding
self.layer.compute_output_shape(self._get_shape(inputs)),
File "/home/de1119151/PycharmProjects/whole-slide-cnn-main/venv/lib/python3.8/site-packages/tensorflow/python/keras/layers/convolutional.py", line 2868, in compute_output_shape
if input_shape[1] is not None:
IndexError: list index out of range

Process finished with exit code 1

from whole-slide-cnn.