2019-09-03 13:02:34.743548: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7fd6a6cd1b00 next 95 of size 512
2019-09-03 13:02:34.743569: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7fd6a6cd1d00 next 96 of size 512
2019-09-03 13:02:34.743590: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7fd6a6cd1f00 next 97 of size 512
2019-09-03 13:02:34.743611: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7fd6a6cd2100 next 98 of size 512
2019-09-03 13:02:34.743632: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7fd6a6cd2300 next 99 of size 512
2019-09-03 13:02:34.743653: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7fd6a6cd2500 next 100 of size 512
2019-09-03 13:02:34.743674: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7fd6a6cd2700 next 101 of size 512
2019-09-03 13:02:34.743696: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7fd6a6cd2900 next 102 of size 512
2019-09-03 13:02:34.743717: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7fd6a6cd2b00 next 103 of size 512
2019-09-03 13:02:34.743738: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7fd6a6cd2d00 next 104 of size 512
2019-09-03 13:02:34.743758: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7fd6a6cd2f00 next 105 of size 512
2019-09-03 13:02:34.743780: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7fd6a6cd3100 next 106 of size 256
2019-09-03 13:02:34.743801: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7fd6a6cd3200 next 107 of size 589824
2019-09-03 13:02:34.743823: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7fd6a6d63200 next 18446744073709551615 of size 642560
2019-09-03 13:02:34.743843: I tensorflow/core/common_runtime/bfc_allocator.cc:793] Next region of size 4194304
2019-09-03 13:02:34.743866: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7fd6a6e00000 next 18446744073709551615 of size 4194304
2019-09-03 13:02:34.743887: I tensorflow/core/common_runtime/bfc_allocator.cc:793] Next region of size 8388608
2019-09-03 13:02:34.743908: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7fd6a7200000 next 71 of size 4194304
2019-09-03 13:02:34.743930: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7fd6a7600000 next 18446744073709551615 of size 4194304
2019-09-03 13:02:34.743950: I tensorflow/core/common_runtime/bfc_allocator.cc:809] Summary of in-use Chunks by size:
2019-09-03 13:02:34.743980: I tensorflow/core/common_runtime/bfc_allocator.cc:812] 868 Chunks of size 256 totalling 217.0KiB
2019-09-03 13:02:34.744004: I tensorflow/core/common_runtime/bfc_allocator.cc:812] 480 Chunks of size 512 totalling 240.0KiB
2019-09-03 13:02:34.744027: I tensorflow/core/common_runtime/bfc_allocator.cc:812] 886 Chunks of size 1024 totalling 886.0KiB
2019-09-03 13:02:34.744050: I tensorflow/core/common_runtime/bfc_allocator.cc:812] 1 Chunks of size 1280 totalling 1.2KiB
2019-09-03 13:02:34.744073: I tensorflow/core/common_runtime/bfc_allocator.cc:812] 567 Chunks of size 2048 totalling 1.11MiB
2019-09-03 13:02:34.744096: I tensorflow/core/common_runtime/bfc_allocator.cc:812] 8 Chunks of size 2816 totalling 22.0KiB
2019-09-03 13:02:34.744119: I tensorflow/core/common_runtime/bfc_allocator.cc:812] 400 Chunks of size 4096 totalling 1.56MiB
2019-09-03 13:02:34.744141: I tensorflow/core/common_runtime/bfc_allocator.cc:812] 190 Chunks of size 8192 totalling 1.48MiB
2019-09-03 13:02:34.744165: I tensorflow/core/common_runtime/bfc_allocator.cc:812] 33 Chunks of size 16384 totalling 528.0KiB
2019-09-03 13:02:34.744187: I tensorflow/core/common_runtime/bfc_allocator.cc:812] 19 Chunks of size 18432 totalling 342.0KiB
2019-09-03 13:02:34.744210: I tensorflow/core/common_runtime/bfc_allocator.cc:812] 33 Chunks of size 37632 totalling 1.18MiB
2019-09-03 13:02:34.744233: I tensorflow/core/common_runtime/bfc_allocator.cc:812] 166 Chunks of size 65536 totalling 10.38MiB
2019-09-03 13:02:34.752035: I tensorflow/core/common_runtime/bfc_allocator.cc:812] 26 Chunks of size 73728 totalling 1.83MiB
2019-09-03 13:02:34.752086: I tensorflow/core/common_runtime/bfc_allocator.cc:812] 33 Chunks of size 131072 totalling 4.12MiB
2019-09-03 13:02:34.752117: I tensorflow/core/common_runtime/bfc_allocator.cc:812] 83 Chunks of size 147456 totalling 11.67MiB
2019-09-03 13:02:34.752145: I tensorflow/core/common_runtime/bfc_allocator.cc:812] 26 Chunks of size 172032 totalling 4.27MiB
2019-09-03 13:02:34.752174: I tensorflow/core/common_runtime/bfc_allocator.cc:812] 191 Chunks of size 262144 totalling 47.75MiB
2019-09-03 13:02:34.752202: I tensorflow/core/common_runtime/bfc_allocator.cc:812] 66 Chunks of size 524288 totalling 33.00MiB
2019-09-03 13:02:34.752223: I tensorflow/core/common_runtime/bfc_allocator.cc:812] 107 Chunks of size 589824 totalling 60.19MiB
2019-09-03 13:02:34.752244: I tensorflow/core/common_runtime/bfc_allocator.cc:812] 1 Chunks of size 642560 totalling 627.5KiB
2019-09-03 13:02:34.752264: I tensorflow/core/common_runtime/bfc_allocator.cc:812] 25 Chunks of size 655360 totalling 15.62MiB
2019-09-03 13:02:34.752289: I tensorflow/core/common_runtime/bfc_allocator.cc:812] 1 Chunks of size 951552 totalling 929.2KiB
2019-09-03 13:02:34.752314: I tensorflow/core/common_runtime/bfc_allocator.cc:812] 290 Chunks of size 1048576 totalling 290.00MiB
2019-09-03 13:02:34.752339: I tensorflow/core/common_runtime/bfc_allocator.cc:812] 1 Chunks of size 1117952 totalling 1.07MiB
2019-09-03 13:02:34.752367: I tensorflow/core/common_runtime/bfc_allocator.cc:812] 53 Chunks of size 2097152 totalling 106.00MiB
2019-09-03 13:02:34.752394: I tensorflow/core/common_runtime/bfc_allocator.cc:812] 158 Chunks of size 2359296 totalling 355.50MiB
2019-09-03 13:02:34.752423: I tensorflow/core/common_runtime/bfc_allocator.cc:812] 1 Chunks of size 3465216 totalling 3.30MiB
2019-09-03 13:02:34.752444: I tensorflow/core/common_runtime/bfc_allocator.cc:812] 1 Chunks of size 4128768 totalling 3.94MiB
2019-09-03 13:02:34.752465: I tensorflow/core/common_runtime/bfc_allocator.cc:812] 82 Chunks of size 4194304 totalling 328.00MiB
2019-09-03 13:02:34.752484: I tensorflow/core/common_runtime/bfc_allocator.cc:812] 1 Chunks of size 7429632 totalling 7.08MiB
2019-09-03 13:02:34.752504: I tensorflow/core/common_runtime/bfc_allocator.cc:812] 19 Chunks of size 8388608 totalling 152.00MiB
2019-09-03 13:02:34.752524: I tensorflow/core/common_runtime/bfc_allocator.cc:812] 45 Chunks of size 9437184 totalling 405.00MiB
2019-09-03 13:02:34.752541: I tensorflow/core/common_runtime/bfc_allocator.cc:812] 1 Chunks of size 9961472 totalling 9.50MiB
2019-09-03 13:02:34.752558: I tensorflow/core/common_runtime/bfc_allocator.cc:812] 12 Chunks of size 10272768 totalling 117.56MiB
2019-09-03 13:02:34.752575: I tensorflow/core/common_runtime/bfc_allocator.cc:812] 1 Chunks of size 14929920 totalling 14.24MiB
2019-09-03 13:02:34.752592: I tensorflow/core/common_runtime/bfc_allocator.cc:812] 1 Chunks of size 15533056 totalling 14.81MiB
2019-09-03 13:02:34.752609: I tensorflow/core/common_runtime/bfc_allocator.cc:812] 1 Chunks of size 15687168 totalling 14.96MiB
2019-09-03 13:02:34.752626: I tensorflow/core/common_runtime/bfc_allocator.cc:812] 17 Chunks of size 18874368 totalling 306.00MiB
2019-09-03 13:02:34.752643: I tensorflow/core/common_runtime/bfc_allocator.cc:812] 1 Chunks of size 18936832 totalling 18.06MiB
2019-09-03 13:02:34.752660: I tensorflow/core/common_runtime/bfc_allocator.cc:812] 6 Chunks of size 20275200 totalling 116.02MiB
2019-09-03 13:02:34.752677: I tensorflow/core/common_runtime/bfc_allocator.cc:812] 1 Chunks of size 21097472 totalling 20.12MiB
2019-09-03 13:02:34.752693: I tensorflow/core/common_runtime/bfc_allocator.cc:812] 1 Chunks of size 30767872 totalling 29.34MiB
2019-09-03 13:02:34.752710: I tensorflow/core/common_runtime/bfc_allocator.cc:812] 1 Chunks of size 33554432 totalling 32.00MiB
2019-09-03 13:02:34.752726: I tensorflow/core/common_runtime/bfc_allocator.cc:812] 1 Chunks of size 40009728 totalling 38.16MiB
2019-09-03 13:02:34.752743: I tensorflow/core/common_runtime/bfc_allocator.cc:812] 7 Chunks of size 40280064 totalling 268.90MiB
2019-09-03 13:02:34.752760: I tensorflow/core/common_runtime/bfc_allocator.cc:812] 22 Chunks of size 41091072 totalling 862.12MiB
2019-09-03 13:02:34.752776: I tensorflow/core/common_runtime/bfc_allocator.cc:812] 1 Chunks of size 42663936 totalling 40.69MiB
2019-09-03 13:02:34.752793: I tensorflow/core/common_runtime/bfc_allocator.cc:812] 1 Chunks of size 50486272 totalling 48.15MiB
2019-09-03 13:02:34.752815: I tensorflow/core/common_runtime/bfc_allocator.cc:812] 1 Chunks of size 70828032 totalling 67.55MiB
2019-09-03 13:02:34.752834: I tensorflow/core/common_runtime/bfc_allocator.cc:812] 3 Chunks of size 81100800 totalling 232.03MiB
2019-09-03 13:02:34.752852: I tensorflow/core/common_runtime/bfc_allocator.cc:812] 3 Chunks of size 161120256 totalling 460.97MiB
2019-09-03 13:02:34.752868: I tensorflow/core/common_runtime/bfc_allocator.cc:812] 1 Chunks of size 162278400 totalling 154.76MiB
2019-09-03 13:02:34.752885: I tensorflow/core/common_runtime/bfc_allocator.cc:812] 1 Chunks of size 201400320 totalling 192.07MiB
2019-09-03 13:02:34.752901: I tensorflow/core/common_runtime/bfc_allocator.cc:816] Sum Total of in-use chunks: 4.79GiB
2019-09-03 13:02:34.752917: I tensorflow/core/common_runtime/bfc_allocator.cc:818] total_region_allocated_bytes_: 5308940288 memory_limit_: 5308940288 available bytes: 0 curr_region_allocation_bytes_: 4294967296
2019-09-03 13:02:34.752942: I tensorflow/core/common_runtime/bfc_allocator.cc:824] Stats:
Limit: 5308940288
InUse: 5146169600
MaxInUse: 5296907776
NumAllocs: 470936928
MaxAllocSize: 2802284544
2019-09-03 13:02:34.753288: W tensorflow/core/common_runtime/bfc_allocator.cc:319] ****************************************************************************************************
2019-09-03 13:02:34.753332: W tensorflow/core/framework/op_kernel.cc:1502] OP_REQUIRES failed at strided_slice_op.cc:246 : Resource exhausted: OOM when allocating tensor with shape[1,38,264,1024] and type float on /job:localhost/replica:0/task:0/device:GPU:0 by allocator GPU_0_bfc
Traceback (most recent call last):
File "/home/kamgo/environments/pyp36/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1356, in _do_call
return fn(*args)
File "/home/kamgo/environments/pyp36/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1341, in _run_fn
options, feed_dict, fetch_list, target_list, run_metadata)
File "/home/kamgo/environments/pyp36/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1429, in _call_tf_sessionrun
run_metadata)
tensorflow.python.framework.errors_impl.ResourceExhaustedError: OOM when allocating tensor with shape[1,38,264,1024] and type float on /job:localhost/replica:0/task:0/device:GPU:0 by allocator GPU_0_bfc
[[{{node gradients_7/roi_pooling_conv_7/strided_slice_99_grad/StridedSliceGrad}}]]
Hint: If you want to see a list of allocated tensors when OOM happens, add report_tensor_allocations_upon_oom to RunOptions for current allocation info.
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "active_learning_modul.py", line 253, in
train_simple()
File "active_learning_modul.py", line 178, in train_simple
con = train.train_model(seed_imgs,seed_classes_count,seed_classes_mapping,con,Earlystopping_patience,config_output_filename)
File "/mnt/0CCCB718CCB6FB52/Projekt/Active-Learning-Faster-RCNN/keras_frcnn/train_frcnn.py", line 230, in train_model
loss_class = model_classifier.train_on_batch([X, X2[:, sel_samples, :]], [Y1[:, sel_samples, :], Y2[:, sel_samples, :]])
File "/home/kamgo/environments/pyp36/lib/python3.6/site-packages/keras/engine/training.py", line 1621, in train_on_batch
outputs = self.train_function(ins)
File "/home/kamgo/environments/pyp36/lib/python3.6/site-packages/keras/backend/tensorflow_backend.py", line 2103, in call
feed_dict=feed_dict)
File "/home/kamgo/environments/pyp36/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 950, in run
run_metadata_ptr)
File "/home/kamgo/environments/pyp36/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1173, in _run
feed_dict_tensor, options, run_metadata)
File "/home/kamgo/environments/pyp36/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1350, in _do_run
run_metadata)
File "/home/kamgo/environments/pyp36/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1370, in _do_call
raise type(e)(node_def, op, message)
tensorflow.python.framework.errors_impl.ResourceExhaustedError: OOM when allocating tensor with shape[1,38,264,1024] and type float on /job:localhost/replica:0/task:0/device:GPU:0 by allocator GPU_0_bfc
[[node gradients_7/roi_pooling_conv_7/strided_slice_99_grad/StridedSliceGrad (defined at /home/kamgo/environments/pyp36/lib/python3.6/site-packages/keras/backend/tensorflow_backend.py:2138) ]]
Hint: If you want to see a list of allocated tensors when OOM happens, add report_tensor_allocations_upon_oom to RunOptions for current allocation info.
Errors may have originated from an input operation.
Input Source operations connected to node gradients_7/roi_pooling_conv_7/strided_slice_99_grad/StridedSliceGrad:
roi_pooling_conv_7/strided_slice_99/stack_2 (defined at /mnt/0CCCB718CCB6FB52/Projekt/Active-Learning-Faster-RCNN/keras_frcnn/RoiPoolingConv.py:105)
Original stack trace for 'gradients_7/roi_pooling_conv_7/strided_slice_99_grad/StridedSliceGrad':
File "active_learning_modul.py", line 253, in
train_simple()
File "active_learning_modul.py", line 178, in train_simple
con = train.train_model(seed_imgs,seed_classes_count,seed_classes_mapping,con,Earlystopping_patience,config_output_filename)
File "/mnt/0CCCB718CCB6FB52/Projekt/Active-Learning-Faster-RCNN/keras_frcnn/train_frcnn.py", line 230, in train_model
loss_class = model_classifier.train_on_batch([X, X2[:, sel_samples, :]], [Y1[:, sel_samples, :], Y2[:, sel_samples, :]])
File "/home/kamgo/environments/pyp36/lib/python3.6/site-packages/keras/engine/training.py", line 1620, in train_on_batch
self._make_train_function()
File "/home/kamgo/environments/pyp36/lib/python3.6/site-packages/keras/engine/training.py", line 1002, in _make_train_function
self.total_loss)
File "/home/kamgo/environments/pyp36/lib/python3.6/site-packages/keras/optimizers.py", line 381, in get_updates
grads = self.get_gradients(loss, params)
File "/home/kamgo/environments/pyp36/lib/python3.6/site-packages/keras/optimizers.py", line 47, in get_gradients
grads = K.gradients(loss, params)
File "/home/kamgo/environments/pyp36/lib/python3.6/site-packages/keras/backend/tensorflow_backend.py", line 2138, in gradients
return tf.gradients(loss, variables, colocate_gradients_with_ops=True)
File "/home/kamgo/environments/pyp36/lib/python3.6/site-packages/tensorflow/python/ops/gradients_impl.py", line 158, in gradients
unconnected_gradients)
File "/home/kamgo/environments/pyp36/lib/python3.6/site-packages/tensorflow/python/ops/gradients_util.py", line 731, in _GradientsHelper
lambda: grad_fn(op, *out_grads))
File "/home/kamgo/environments/pyp36/lib/python3.6/site-packages/tensorflow/python/ops/gradients_util.py", line 403, in _MaybeCompile
return grad_fn() # Exit early
File "/home/kamgo/environments/pyp36/lib/python3.6/site-packages/tensorflow/python/ops/gradients_util.py", line 731, in
lambda: grad_fn(op, *out_grads))
File "/home/kamgo/environments/pyp36/lib/python3.6/site-packages/tensorflow/python/ops/array_grad.py", line 279, in _StridedSliceGrad
shrink_axis_mask=op.get_attr("shrink_axis_mask")), None, None, None
File "/home/kamgo/environments/pyp36/lib/python3.6/site-packages/tensorflow/python/ops/gen_array_ops.py", line 10193, in strided_slice_grad
shrink_axis_mask=shrink_axis_mask, name=name)
File "/home/kamgo/environments/pyp36/lib/python3.6/site-packages/tensorflow/python/framework/op_def_library.py", line 788, in _apply_op_helper
op_def=op_def)
File "/home/kamgo/environments/pyp36/lib/python3.6/site-packages/tensorflow/python/util/deprecation.py", line 507, in new_func
return func(*args, **kwargs)
File "/home/kamgo/environments/pyp36/lib/python3.6/site-packages/tensorflow/python/framework/ops.py", line 3616, in create_op
op_def=op_def)
File "/home/kamgo/environments/pyp36/lib/python3.6/site-packages/tensorflow/python/framework/ops.py", line 2005, in init
self._traceback = tf_stack.extract_stack()
...which was originally created as op 'roi_pooling_conv_7/strided_slice_99', defined at:
File "active_learning_modul.py", line 253, in
train_simple()
[elided 0 identical lines from previous traceback]
File "active_learning_modul.py", line 178, in train_simple
con = train.train_model(seed_imgs,seed_classes_count,seed_classes_mapping,con,Earlystopping_patience,config_output_filename)
File "/mnt/0CCCB718CCB6FB52/Projekt/Active-Learning-Faster-RCNN/keras_frcnn/train_frcnn.py", line 106, in train_model
classifier = nn.classifier(shared_layers, roi_input, con.num_rois, nb_classes=len(classes_count), trainable=True)
File "/mnt/0CCCB718CCB6FB52/Projekt/Active-Learning-Faster-RCNN/keras_frcnn/resnet.py", line 239, in classifier
out_roi_pool = RoiPoolingConv(pooling_regions, num_rois)([base_layers, input_rois])
File "/home/kamgo/environments/pyp36/lib/python3.6/site-packages/keras/engine/topology.py", line 578, in call
output = self.call(inputs, **kwargs)
File "/mnt/0CCCB718CCB6FB52/Projekt/Active-Learning-Faster-RCNN/keras_frcnn/RoiPoolingConv.py", line 105, in call
rs = tf.image.resize_images(img[:, y:y+h, x:x+w, :], (self.pool_size, self.pool_size))
File "/home/kamgo/environments/pyp36/lib/python3.6/site-packages/tensorflow/python/ops/array_ops.py", line 680, in _slice_helper
name=name)
File "/home/kamgo/environments/pyp36/lib/python3.6/site-packages/tensorflow/python/ops/array_ops.py", line 846, in strided_slice
shrink_axis_mask=shrink_axis_mask)
File "/home/kamgo/environments/pyp36/lib/python3.6/site-packages/tensorflow/python/ops/gen_array_ops.py", line 9989, in strided_slice
shrink_axis_mask=shrink_axis_mask, name=name)
File "/home/kamgo/environments/pyp36/lib/python3.6/site-packages/tensorflow/python/framework/op_def_library.py", line 788, in _apply_op_helper
op_def=op_def)
File "/home/kamgo/environments/pyp36/lib/python3.6/site-packages/tensorflow/python/util/deprecation.py", line 507, in new_func
return func(*args, **kwargs)
File "/home/kamgo/environments/pyp36/lib/python3.6/site-packages/tensorflow/python/framework/ops.py", line 3616, in create_op
op_def=op_def)
File "/home/kamgo/environments/pyp36/lib/python3.6/site-packages/tensorflow/python/framework/ops.py", line 2005, in init
self._traceback = tf_stack.extract_stack()
I not understand why i receive that error during training.
I won say that I use the repository to implement active learning. that means i loop on data to train the model. the train function doesn't give error during the tree first iteration and then i receive this error.
please help me !
React
Typescript
Django
Laravel
Facebook
Microsoft
Google
Alibaba
D3
Tencent