Comments (2)
To be sure, I show the full text of the error.
(HOnnotate) initial@d3050eb4fa4f:~/workspace/HOnnotate/optimization$ python objectTrackingSingleFrame.py --seq 'test'
/home/initial/miniconda3/envs/HOnnotate/lib/python3.5/site-packages/IPython/lib/pretty.py:91: DeprecationWarning: IPython.utils.signatures backport for Python 2 is deprecated in IPython 6, which only supports Python 3
from IPython.utils.signatures import signature
/home/initial/miniconda3/envs/HOnnotate/lib/python3.5/site-packages/tensorflow/python/util/tf_inspect.py:75: DeprecationWarning: inspect.getargspec() is deprecated, use inspect.signature() instead
return _inspect.getargspec(target)
/home/initial/miniconda3/envs/HOnnotate/lib/python3.5/site-packages/tensorflow/python/util/tf_inspect.py:75: DeprecationWarning: inspect.getargspec() is deprecated, use inspect.signature() instead
return _inspect.getargspec(target)
/home/initial/miniconda3/envs/HOnnotate/lib/python3.5/site-packages/tensorflow/python/util/tf_inspect.py:75: DeprecationWarning: inspect.getargspec() is deprecated, use inspect.signature() instead
return _inspect.getargspec(target)
/home/initial/miniconda3/envs/HOnnotate/lib/python3.5/site-packages/tensorflow/python/util/tf_inspect.py:75: DeprecationWarning: inspect.getargspec() is deprecated, use inspect.signature() instead
return _inspect.getargspec(target)
/home/initial/miniconda3/envs/HOnnotate/lib/python3.5/site-packages/tensorflow/python/util/tf_inspect.py:75: DeprecationWarning: inspect.getargspec() is deprecated, use inspect.signature() instead
return _inspect.getargspec(target)
WARNING:tensorflow:From /home/initial/workspace/HOnnotate/optimization/ghope/loss.py:225: Print (from tensorflow.python.ops.logging_ops) is deprecated and will be removed after 2018-08-20.
Instructions for updating:
Use tf.print instead of tf.Print. Note that tf.print returns a no-output operator that directly prints the output. Outside of defuns or eager mode, this operator will not be executed unless it is directly specified in session.run or used as a control dependency for other operators. This is only a concern in graph mode. Below is an example of how to ensure tf.print executes in graph mode:
sess = tf.Session()
with sess.as_default():
tensor = tf.range(10)
print_op = tf.print(tensor)
with tf.control_dependencies([print_op]):
out = tf.add(tensor, tensor)
sess.run(out)
```
Additionally, to use tf.print in python 2.7, users must make sure to import
the following:
`from __future__ import print_function`
/home/initial/miniconda3/envs/HOnnotate/lib/python3.5/site-packages/chumpy/ch.py:1203: DeprecationWarning: inspect.getargspec() is deprecated, use inspect.signature() instead
want_out = 'out' in inspect.getargspec(func).args
/home/initial/miniconda3/envs/HOnnotate/lib/python3.5/site-packages/numpy/matrixlib/defmatrix.py:68: PendingDeprecationWarning: the matrix subclass is not the recommended way to represent matrices or deal with linear algebra (see https://docs.scipy.org/doc/numpy/user/numpy-for-matlab-users.html). Please adjust your code to use regular ndarray.
return matrix(data, dtype=dtype, copy=False)
/home/initial/workspace/HOnnotate/optimization/object/batch_object.py:65: RuntimeWarning: invalid value encountered in true_divide
vn = mesh.vn / np.expand_dims(np.linalg.norm(mesh.vn, ord=2, axis=1), 1) # normalize to unit vec
/home/initial/workspace/HOnnotate/optimization/ghope/loss.py:106: DeprecationWarning: Both axis > a.ndim and axis < -a.ndim - 1 are deprecated and will raise an AxisError in the future.
gaussKernel = np.expand_dims(gaussKernel, 3)
/home/initial/workspace/HOnnotate/optimization/ghope/loss.py:106: DeprecationWarning: Both axis > a.ndim and axis < -a.ndim - 1 are deprecated and will raise an AxisError in the future.
gaussKernel = np.expand_dims(gaussKernel, 3)
/home/initial/workspace/HOnnotate/optimization/ghope/loss.py:106: DeprecationWarning: Both axis > a.ndim and axis < -a.ndim - 1 are deprecated and will raise an AxisError in the future.
gaussKernel = np.expand_dims(gaussKernel, 3)
/home/initial/workspace/HOnnotate/optimization/ghope/loss.py:106: DeprecationWarning: Both axis > a.ndim and axis < -a.ndim - 1 are deprecated and will raise an AxisError in the future.
gaussKernel = np.expand_dims(gaussKernel, 3)
WARNING:tensorflow:From /home/initial/workspace/HOnnotate/optimization/ghope/icp.py:112: arg_min (from tensorflow.python.ops.gen_math_ops) is deprecated and will be removed in a future version.
Instructions for updating:
Use `argmin` instead
2024-01-14 10:50:00.623402: I tensorflow/core/platform/cpu_feature_guard.cc:141] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2 FMA
2024-01-14 10:50:00.999041: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1432] Found device 0 with properties:
name: Quadro P5000 major: 6 minor: 1 memoryClockRate(GHz): 1.7335
pciBusID: 0000:03:00.0
totalMemory: 15.90GiB freeMemory: 15.78GiB
2024-01-14 10:50:01.314176: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1432] Found device 1 with properties:
name: NVIDIA GeForce RTX 3090 major: 8 minor: 6 memoryClockRate(GHz): 1.725
pciBusID: 0000:02:00.0
totalMemory: 23.70GiB freeMemory: 22.76GiB
2024-01-14 10:50:01.314243: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1511] Adding visible gpu devices: 0, 1
2024-01-14 10:50:28.016592: I tensorflow/core/common_runtime/gpu/gpu_device.cc:982] Device interconnect StreamExecutor with strength 1 edge matrix:
2024-01-14 10:50:28.016640: I tensorflow/core/common_runtime/gpu/gpu_device.cc:988] 0 1
2024-01-14 10:50:28.016651: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1001] 0: N N
2024-01-14 10:50:28.016657: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1001] 1: N N
2024-01-14 10:50:28.016832: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1115] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 15293 MB memory) -> physical GPU (device: 0, name: Quadro P5000, pci bus id: 0000:03:00.0, compute capability: 6.1)
2024-01-14 10:50:28.033847: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1115] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:1 with 22064 MB memory) -> physical GPU (device: 1, name: NVIDIA GeForce RTX 3090, pci bus id: 0000:02:00.0, compute capability: 8.6)
Clipping input data to the valid range for imshow with RGB data ([0..1] for floats or [0..255] for integers).
maskPC for Image test/0/00002 is 0.132587
[Loading New frame ][0/00002]
2024-01-14 10:50:47.882958: W tensorflow/core/common_runtime/bfc_allocator.cc:267] Allocator (GPU_0_bfc) ran out of memory trying to allocate 29.75GiB. Current allocation summary follows.
2024-01-14 10:50:47.883086: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (256): Total Chunks: 68, Chunks in use: 66. 17.0KiB allocated for chunks. 16.5KiB in use in bin. 788B client-requested in use in bin.
2024-01-14 10:50:47.883121: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (512): Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2024-01-14 10:50:47.883151: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (1024): Total Chunks: 1, Chunks in use: 1. 1.2KiB allocated for chunks. 1.2KiB in use in bin. 1.0KiB client-requested in use in bin.
2024-01-14 10:50:47.883177: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (2048): Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2024-01-14 10:50:47.883202: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (4096): Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2024-01-14 10:50:47.883229: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (8192): Total Chunks: 2, Chunks in use: 2. 18.5KiB allocated for chunks. 18.5KiB in use in bin. 18.2KiB client-requested in use in bin.
2024-01-14 10:50:47.883253: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (16384): Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2024-01-14 10:50:47.883281: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (32768): Total Chunks: 2, Chunks in use: 2. 97.5KiB allocated for chunks. 97.5KiB in use in bin. 97.2KiB client-requested in use in bin.
2024-01-14 10:50:47.883309: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (65536): Total Chunks: 3, Chunks in use: 3. 301.8KiB allocated for chunks. 301.8KiB in use in bin. 301.4KiB client-requested in use in bin.
2024-01-14 10:50:47.883358: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (131072): Total Chunks: 1, Chunks in use: 0. 134.8KiB allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2024-01-14 10:50:47.883390: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (262144): Total Chunks: 3, Chunks in use: 3. 1014.2KiB allocated for chunks. 1014.2KiB in use in bin. 900.0KiB client-requested in use in bin.
2024-01-14 10:50:47.883417: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (524288): Total Chunks: 10, Chunks in use: 10. 8.79MiB allocated for chunks. 8.79MiB in use in bin. 8.79MiB client-requested in use in bin.
2024-01-14 10:50:47.883442: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (1048576): Total Chunks: 4, Chunks in use: 3. 5.17MiB allocated for chunks. 4.00MiB in use in bin. 3.84MiB client-requested in use in bin.
2024-01-14 10:50:47.883468: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (2097152): Total Chunks: 2, Chunks in use: 2. 5.61MiB allocated for chunks. 5.61MiB in use in bin. 4.20MiB client-requested in use in bin.
2024-01-14 10:50:47.883495: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (4194304): Total Chunks: 5, Chunks in use: 3. 23.89MiB allocated for chunks. 14.00MiB in use in bin. 14.00MiB client-requested in use in bin.
2024-01-14 10:50:47.883525: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (8388608): Total Chunks: 2, Chunks in use: 2. 24.00MiB allocated for chunks. 24.00MiB in use in bin. 20.00MiB client-requested in use in bin.
2024-01-14 10:50:47.883550: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (16777216): Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2024-01-14 10:50:47.883575: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (33554432): Total Chunks: 1, Chunks in use: 0. 58.00MiB allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2024-01-14 10:50:47.883600: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (67108864): Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2024-01-14 10:50:47.883625: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (134217728): Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2024-01-14 10:50:47.883649: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (268435456): Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2024-01-14 10:50:47.883675: I tensorflow/core/common_runtime/bfc_allocator.cc:613] Bin for 29.75GiB was 256.00MiB, Chunk State:
2024-01-14 10:50:47.883700: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7fc7d0000000 of size 6291712
2024-01-14 10:50:47.883722: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Free at 0x7fc7d0600100 of size 60817152
2024-01-14 10:50:47.883743: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7fc7e6000000 of size 12582912
2024-01-14 10:50:47.883763: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Free at 0x7fc7e6c00000 of size 4194560
2024-01-14 10:50:47.883784: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7fc7e7000100 of size 4194560
2024-01-14 10:50:47.883805: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7fc7e7400200 of size 12582400
2024-01-14 10:50:47.883825: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7fc7ea800000 of size 307200
2024-01-14 10:50:47.883845: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7fc7ea84b000 of size 256
2024-01-14 10:50:47.883865: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7fc7ea84b100 of size 4194560
2024-01-14 10:50:47.883885: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7fc7eac4b200 of size 256
2024-01-14 10:50:47.883905: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7fc7eac4b300 of size 256
2024-01-14 10:50:47.883924: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7fc7eac4b400 of size 256
2024-01-14 10:50:47.883944: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7fc7eac4b500 of size 256
2024-01-14 10:50:47.883963: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7fc7eac4b600 of size 256
2024-01-14 10:50:47.883982: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7fc7eac4b700 of size 256
2024-01-14 10:50:47.884001: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7fc7eac4b800 of size 256
2024-01-14 10:50:47.884021: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7fc7eac4b900 of size 256
2024-01-14 10:50:47.884040: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7fc7eac4ba00 of size 921600
2024-01-14 10:50:47.884060: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7fc7ead2ca00 of size 256
2024-01-14 10:50:47.884079: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7fc7ead2cb00 of size 256
2024-01-14 10:50:47.884098: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7fc7ead2cc00 of size 921600
2024-01-14 10:50:47.884117: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7fc7eae0dc00 of size 921600
2024-01-14 10:50:47.884136: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7fc7eaeeec00 of size 256
2024-01-14 10:50:47.884155: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7fc7eaeeed00 of size 256
2024-01-14 10:50:47.884174: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7fc7eaeeee00 of size 256
2024-01-14 10:50:47.884193: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7fc7eaeeef00 of size 256
2024-01-14 10:50:47.884213: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7fc7eaeef000 of size 256
2024-01-14 10:50:47.884233: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Free at 0x7fc7eaeef100 of size 256
2024-01-14 10:50:47.884251: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7fc7eaeef200 of size 256
2024-01-14 10:50:47.884270: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7fc7eaeef300 of size 256
2024-01-14 10:50:47.884289: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7fc7eaeef400 of size 256
2024-01-14 10:50:47.884308: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7fc7eaeef500 of size 256
2024-01-14 10:50:47.884327: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7fc7eaeef600 of size 256
2024-01-14 10:50:47.884346: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7fc7eaeef700 of size 256
2024-01-14 10:50:47.884365: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7fc7eaeef800 of size 256
2024-01-14 10:50:47.884384: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7fc7eaeef900 of size 256
2024-01-14 10:50:47.884403: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7fc7eaeefa00 of size 256
2024-01-14 10:50:47.884421: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Free at 0x7fc7eaeefb00 of size 256
2024-01-14 10:50:47.884440: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7fc7eaeefc00 of size 256
2024-01-14 10:50:47.884459: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7fc7eaeefd00 of size 921600
2024-01-14 10:50:47.884479: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Free at 0x7fc7eafd0d00 of size 1228800
2024-01-14 10:50:47.884498: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7fc7eb0fcd00 of size 921600
2024-01-14 10:50:47.884517: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Free at 0x7fc7eb1ddd00 of size 137984
2024-01-14 10:50:47.884536: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7fc7eb1ff800 of size 122112
2024-01-14 10:50:47.884556: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Free at 0x7fc7eb21d500 of size 6171392
2024-01-14 10:50:47.884576: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7fc8f2a00000 of size 9472
2024-01-14 10:50:47.884596: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7fc8f2a02500 of size 256
2024-01-14 10:50:47.884615: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7fc8f2a02600 of size 256
2024-01-14 10:50:47.884635: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7fc8f2a02700 of size 93440
2024-01-14 10:50:47.884654: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7fc8f2a19400 of size 256
2024-01-14 10:50:47.884674: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7fc8f2a19500 of size 49920
2024-01-14 10:50:47.884693: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7fc8f2a25800 of size 256
2024-01-14 10:50:47.884712: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7fc8f2a25900 of size 256
2024-01-14 10:50:47.884731: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7fc8f2a25a00 of size 256
2024-01-14 10:50:47.884751: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7fc8f2a25b00 of size 1280
2024-01-14 10:50:47.884771: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7fc8f2a26000 of size 9472
2024-01-14 10:50:47.884790: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7fc8f2a28500 of size 256
2024-01-14 10:50:47.884810: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7fc8f2a28600 of size 256
2024-01-14 10:50:47.884829: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7fc8f2a28700 of size 256
2024-01-14 10:50:47.884848: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7fc8f2a28800 of size 256
2024-01-14 10:50:47.884867: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7fc8f2a28900 of size 256
2024-01-14 10:50:47.884887: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7fc8f2a28a00 of size 256
2024-01-14 10:50:47.884906: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7fc8f2a28b00 of size 93440
2024-01-14 10:50:47.884925: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7fc8f2a3f800 of size 256
2024-01-14 10:50:47.884944: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7fc8f2a3f900 of size 49920
2024-01-14 10:50:47.884963: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7fc8f2a4bc00 of size 256
2024-01-14 10:50:47.884983: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7fc8f2a4bd00 of size 256
2024-01-14 10:50:47.885002: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7fc8f2a4be00 of size 256
2024-01-14 10:50:47.885021: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7fc8f2a4bf00 of size 256
2024-01-14 10:50:47.885040: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7fc8f2a4c000 of size 256
2024-01-14 10:50:47.885059: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7fc8f2a4c100 of size 256
2024-01-14 10:50:47.885078: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7fc8f2a4c200 of size 256
2024-01-14 10:50:47.885098: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7fc8f2a4c300 of size 256
2024-01-14 10:50:47.885118: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7fc8f2a4c400 of size 256
2024-01-14 10:50:47.885138: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7fc8f2a4c500 of size 256
2024-01-14 10:50:47.885157: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7fc8f2a4c600 of size 256
2024-01-14 10:50:47.885177: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7fc8f2a4c700 of size 256
2024-01-14 10:50:47.885196: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7fc8f2a4c800 of size 256
2024-01-14 10:50:47.885215: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7fc8f2a4c900 of size 256
2024-01-14 10:50:47.885235: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7fc8f2a4ca00 of size 256
2024-01-14 10:50:47.885253: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7fc8f2a4cb00 of size 256
2024-01-14 10:50:47.885272: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7fc8f2a4cc00 of size 256
2024-01-14 10:50:47.885292: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7fc8f2a4cd00 of size 256
2024-01-14 10:50:47.885311: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7fc8f2a4ce00 of size 256
2024-01-14 10:50:47.885330: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7fc8f2a4cf00 of size 256
2024-01-14 10:50:47.885349: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7fc8f2a4d000 of size 256
2024-01-14 10:50:47.885368: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7fc8f2a4d100 of size 256
2024-01-14 10:50:47.885387: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7fc8f2a4d200 of size 256
2024-01-14 10:50:47.885406: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7fc8f2a4d300 of size 256
2024-01-14 10:50:47.885424: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7fc8f2a4d400 of size 256
2024-01-14 10:50:47.885444: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7fc8f2a4d500 of size 256
2024-01-14 10:50:47.885464: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7fc8f2a4d600 of size 256
2024-01-14 10:50:47.885482: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7fc8f2a4d700 of size 307200
2024-01-14 10:50:47.885502: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7fc8f2a98700 of size 424192
2024-01-14 10:50:47.885523: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7fc8f2e00000 of size 2097152
2024-01-14 10:50:47.885543: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7fc8f3000000 of size 921600
2024-01-14 10:50:47.885563: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7fc8f30e1000 of size 1260544
2024-01-14 10:50:47.885582: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7fc8f3214c00 of size 921600
2024-01-14 10:50:47.885602: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7fc8f32f5c00 of size 1090560
2024-01-14 10:50:47.885622: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7fc8f3400000 of size 921600
2024-01-14 10:50:47.885640: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7fc8f34e1000 of size 921600
2024-01-14 10:50:47.885659: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7fc8f35c2000 of size 921600
2024-01-14 10:50:47.885679: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7fc8f36a3000 of size 1843200
2024-01-14 10:50:47.885700: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7fc8f3865000 of size 3780608
2024-01-14 10:50:47.885720: I tensorflow/core/common_runtime/bfc_allocator.cc:638] Summary of in-use Chunks by size:
2024-01-14 10:50:47.885745: I tensorflow/core/common_runtime/bfc_allocator.cc:641] 66 Chunks of size 256 totalling 16.5KiB
2024-01-14 10:50:47.885769: I tensorflow/core/common_runtime/bfc_allocator.cc:641] 1 Chunks of size 1280 totalling 1.2KiB
2024-01-14 10:50:47.885791: I tensorflow/core/common_runtime/bfc_allocator.cc:641] 2 Chunks of size 9472 totalling 18.5KiB
2024-01-14 10:50:47.885813: I tensorflow/core/common_runtime/bfc_allocator.cc:641] 2 Chunks of size 49920 totalling 97.5KiB
2024-01-14 10:50:47.885836: I tensorflow/core/common_runtime/bfc_allocator.cc:641] 2 Chunks of size 93440 totalling 182.5KiB
2024-01-14 10:50:47.885858: I tensorflow/core/common_runtime/bfc_allocator.cc:641] 1 Chunks of size 122112 totalling 119.2KiB
2024-01-14 10:50:47.885880: I tensorflow/core/common_runtime/bfc_allocator.cc:641] 2 Chunks of size 307200 totalling 600.0KiB
2024-01-14 10:50:47.885901: I tensorflow/core/common_runtime/bfc_allocator.cc:641] 1 Chunks of size 424192 totalling 414.2KiB
2024-01-14 10:50:47.885923: I tensorflow/core/common_runtime/bfc_allocator.cc:641] 10 Chunks of size 921600 totalling 8.79MiB
2024-01-14 10:50:47.885945: I tensorflow/core/common_runtime/bfc_allocator.cc:641] 1 Chunks of size 1090560 totalling 1.04MiB
2024-01-14 10:50:47.885966: I tensorflow/core/common_runtime/bfc_allocator.cc:641] 1 Chunks of size 1260544 totalling 1.20MiB
2024-01-14 10:50:47.885987: I tensorflow/core/common_runtime/bfc_allocator.cc:641] 1 Chunks of size 1843200 totalling 1.76MiB
2024-01-14 10:50:47.886008: I tensorflow/core/common_runtime/bfc_allocator.cc:641] 1 Chunks of size 2097152 totalling 2.00MiB
2024-01-14 10:50:47.886029: I tensorflow/core/common_runtime/bfc_allocator.cc:641] 1 Chunks of size 3780608 totalling 3.61MiB
2024-01-14 10:50:47.886050: I tensorflow/core/common_runtime/bfc_allocator.cc:641] 2 Chunks of size 4194560 totalling 8.00MiB
2024-01-14 10:50:47.886071: I tensorflow/core/common_runtime/bfc_allocator.cc:641] 1 Chunks of size 6291712 totalling 6.00MiB
2024-01-14 10:50:47.886094: I tensorflow/core/common_runtime/bfc_allocator.cc:641] 1 Chunks of size 12582400 totalling 12.00MiB
2024-01-14 10:50:47.886116: I tensorflow/core/common_runtime/bfc_allocator.cc:641] 1 Chunks of size 12582912 totalling 12.00MiB
2024-01-14 10:50:47.886138: I tensorflow/core/common_runtime/bfc_allocator.cc:645] Sum Total of in-use chunks: 57.81MiB
2024-01-14 10:50:47.886165: I tensorflow/core/common_runtime/bfc_allocator.cc:647] Stats:
Limit: 16035977626
InUse: 60618752
MaxInUse: 64813312
NumAllocs: 139
MaxAllocSize: 12582912
2024-01-14 10:50:47.886214: W tensorflow/core/common_runtime/bfc_allocator.cc:271] *****_____________________________________________**********__***********xx*********____************
2024-01-14 10:50:47.901539: W tensorflow/core/framework/op_kernel.cc:1273] OP_REQUIRES failed at cwise_ops_common.cc:70 : Resource exhausted: OOM when allocating tensor with shape[7617,262146,4] and type float on /job:localhost/replica:0/task:0/device:GPU:0 by allocator GPU_0_bfc
2024-01-14 10:50:48.043748: I /home/initial/workspace/thirdparty_for_HOnnotate/dirt/csrc/gl_common.h:66] selected egl device #1 to match cuda device #0 for thread 0x7fc7e5fff700
2024-01-14 10:50:48.303976: I /home/initial/workspace/thirdparty_for_HOnnotate/dirt/csrc/gl_common.h:84] successfully created new GL context on thread 0x7fc7e5fff700 (EGL = 1.5, GL = 4.6.0 NVIDIA 470.223.02, renderer = Quadro P5000/PCIe/SSE2)
2024-01-14 10:50:48.664465: I /home/initial/workspace/thirdparty_for_HOnnotate/dirt/csrc/rasterise_egl.cpp:266] reinitialised framebuffer with size 320 x 480
2024-01-14 10:50:48.726655: I /home/initial/workspace/thirdparty_for_HOnnotate/dirt/csrc/gl_common.h:66] selected egl device #1 to match cuda device #0 for thread 0x7fc7e57fe700
2024-01-14 10:50:48.749039: I /home/initial/workspace/thirdparty_for_HOnnotate/dirt/csrc/gl_common.h:84] successfully created new GL context on thread 0x7fc7e57fe700 (EGL = 1.5, GL = 4.6.0 NVIDIA 470.223.02, renderer = Quadro P5000/PCIe/SSE2)
2024-01-14 10:50:48.751036: I /home/initial/workspace/thirdparty_for_HOnnotate/dirt/csrc/rasterise_grad_egl.cpp:314] reinitialised framebuffer with size 320 x 480
Traceback (most recent call last):
File "/home/initial/miniconda3/envs/HOnnotate/lib/python3.5/site-packages/tensorflow/python/client/session.py", line 1334, in _do_call
return fn(*args)
File "/home/initial/miniconda3/envs/HOnnotate/lib/python3.5/site-packages/tensorflow/python/client/session.py", line 1319, in _run_fn
options, feed_dict, fetch_list, target_list, run_metadata)
File "/home/initial/miniconda3/envs/HOnnotate/lib/python3.5/site-packages/tensorflow/python/client/session.py", line 1407, in _call_tf_sessionrun
run_metadata)
tensorflow.python.framework.errors_impl.ResourceExhaustedError: OOM when allocating tensor with shape[7617,262146,4] and type float on /job:localhost/replica:0/task:0/device:GPU:0 by allocator GPU_0_bfc
[[{{node map/while/sub_1}} = Sub[T=DT_FLOAT, _device="/job:localhost/replica:0/task:0/device:GPU:0"](map/while/ExpandDims, map/while/TensorArrayReadV3)]]
Hint: If you want to see a list of allocated tensors when OOM happens, add report_tensor_allocations_upon_oom to RunOptions for current allocation info.
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "objectTrackingSingleFrame.py", line 358, in <module>
objectTracker(w, h, paramInit, camProp, mesh, out_dir, configData)
File "objectTrackingSingleFrame.py", line 156, in objectTracker
opti1.runOptimization(session, 1, {loadData:True})
File "/home/initial/workspace/HOnnotate/optimization/ghope/utils.py", line 159, in timed
result = method(*args, **kw)
File "/home/initial/workspace/HOnnotate/optimization/ghope/optimization.py", line 68, in runOptimization
session.run(self.optOp, feed_dict=feedDict)
File "/home/initial/miniconda3/envs/HOnnotate/lib/python3.5/site-packages/tensorflow/python/client/session.py", line 929, in run
run_metadata_ptr)
File "/home/initial/miniconda3/envs/HOnnotate/lib/python3.5/site-packages/tensorflow/python/client/session.py", line 1152, in _run
feed_dict_tensor, options, run_metadata)
File "/home/initial/miniconda3/envs/HOnnotate/lib/python3.5/site-packages/tensorflow/python/client/session.py", line 1328, in _do_run
run_metadata)
File "/home/initial/miniconda3/envs/HOnnotate/lib/python3.5/site-packages/tensorflow/python/client/session.py", line 1348, in _do_call
raise type(e)(node_def, op, message)
tensorflow.python.framework.errors_impl.ResourceExhaustedError: OOM when allocating tensor with shape[7617,262146,4] and type float on /job:localhost/replica:0/task:0/device:GPU:0 by allocator GPU_0_bfc
[[node map/while/sub_1 (defined at /home/initial/workspace/HOnnotate/optimization/ghope/icp.py:112) = Sub[T=DT_FLOAT, _device="/job:localhost/replica:0/task:0/device:GPU:0"](map/while/ExpandDims, map/while/TensorArrayReadV3)]]
Hint: If you want to see a list of allocated tensors when OOM happens, add report_tensor_allocations_upon_oom to RunOptions for current allocation info.
Caused by op 'map/while/sub_1', defined at:
File "objectTrackingSingleFrame.py", line 358, in <module>
objectTracker(w, h, paramInit, camProp, mesh, out_dir, configData)
File "objectTrackingSingleFrame.py", line 92, in objectTracker
icpLoss = icp.getLoss(finalMesh.vUnClipped)
File "/home/initial/workspace/HOnnotate/optimization/ghope/icp.py", line 191, in getLoss
loss = tf.map_fn(self.getLossPerFrame, (finalMesh, self.pcl2DAllViews, self.realObservs.seg, segColor), parallel_iterations=int(self.numViews), dtype=tf.float32)
File "/home/initial/miniconda3/envs/HOnnotate/lib/python3.5/site-packages/tensorflow/python/ops/functional_ops.py", line 494, in map_fn
maximum_iterations=n)
File "/home/initial/miniconda3/envs/HOnnotate/lib/python3.5/site-packages/tensorflow/python/ops/control_flow_ops.py", line 3291, in while_loop
return_same_structure)
File "/home/initial/miniconda3/envs/HOnnotate/lib/python3.5/site-packages/tensorflow/python/ops/control_flow_ops.py", line 3004, in BuildLoop
pred, body, original_loop_vars, loop_vars, shape_invariants)
File "/home/initial/miniconda3/envs/HOnnotate/lib/python3.5/site-packages/tensorflow/python/ops/control_flow_ops.py", line 2939, in _BuildLoop
body_result = body(*packed_vars_for_body)
File "/home/initial/miniconda3/envs/HOnnotate/lib/python3.5/site-packages/tensorflow/python/ops/control_flow_ops.py", line 3260, in <lambda>
body = lambda i, lv: (i + 1, orig_body(*lv))
File "/home/initial/miniconda3/envs/HOnnotate/lib/python3.5/site-packages/tensorflow/python/ops/functional_ops.py", line 483, in compute
packed_fn_values = fn(packed_values)
File "/home/initial/workspace/HOnnotate/optimization/ghope/icp.py", line 112, in getLossPerFrame
tf.arg_min(tf.reduce_sum(tf.abs(pclValid - verts), axis=2), dimension=1)) # N,
File "/home/initial/miniconda3/envs/HOnnotate/lib/python3.5/site-packages/tensorflow/python/ops/math_ops.py", line 866, in binary_op_wrapper
return func(x, y, name=name)
File "/home/initial/miniconda3/envs/HOnnotate/lib/python3.5/site-packages/tensorflow/python/ops/gen_math_ops.py", line 8318, in sub
"Sub", x=x, y=y, name=name)
File "/home/initial/miniconda3/envs/HOnnotate/lib/python3.5/site-packages/tensorflow/python/framework/op_def_library.py", line 787, in _apply_op_helper
op_def=op_def)
File "/home/initial/miniconda3/envs/HOnnotate/lib/python3.5/site-packages/tensorflow/python/util/deprecation.py", line 488, in new_func
return func(*args, **kwargs)
File "/home/initial/miniconda3/envs/HOnnotate/lib/python3.5/site-packages/tensorflow/python/framework/ops.py", line 3274, in create_op
op_def=op_def)
File "/home/initial/miniconda3/envs/HOnnotate/lib/python3.5/site-packages/tensorflow/python/framework/ops.py", line 1770, in __init__
self._traceback = tf_stack.extract_stack()
ResourceExhaustedError (see above for traceback): OOM when allocating tensor with shape[7617,262146,4] and type float on /job:localhost/replica:0/task:0/device:GPU:0 by allocator GPU_0_bfc
[[node map/while/sub_1 (defined at /home/initial/workspace/HOnnotate/optimization/ghope/icp.py:112) = Sub[T=DT_FLOAT, _device="/job:localhost/replica:0/task:0/device:GPU:0"](map/while/ExpandDims, map/while/TensorArrayReadV3)]]
Hint: If you want to see a list of allocated tensors when OOM happens, add report_tensor_allocations_upon_oom to RunOptions for current allocation info.
from honnotate.
To be sure, I show the full text of the error.
(HOnnotate) initial@d3050eb4fa4f:~/workspace/HOnnotate/optimization$ python objectTrackingSingleFrame.py --seq 'test' /home/initial/miniconda3/envs/HOnnotate/lib/python3.5/site-packages/IPython/lib/pretty.py:91: DeprecationWarning: IPython.utils.signatures backport for Python 2 is deprecated in IPython 6, which only supports Python 3 from IPython.utils.signatures import signature /home/initial/miniconda3/envs/HOnnotate/lib/python3.5/site-packages/tensorflow/python/util/tf_inspect.py:75: DeprecationWarning: inspect.getargspec() is deprecated, use inspect.signature() instead return _inspect.getargspec(target) /home/initial/miniconda3/envs/HOnnotate/lib/python3.5/site-packages/tensorflow/python/util/tf_inspect.py:75: DeprecationWarning: inspect.getargspec() is deprecated, use inspect.signature() instead return _inspect.getargspec(target) /home/initial/miniconda3/envs/HOnnotate/lib/python3.5/site-packages/tensorflow/python/util/tf_inspect.py:75: DeprecationWarning: inspect.getargspec() is deprecated, use inspect.signature() instead return _inspect.getargspec(target) /home/initial/miniconda3/envs/HOnnotate/lib/python3.5/site-packages/tensorflow/python/util/tf_inspect.py:75: DeprecationWarning: inspect.getargspec() is deprecated, use inspect.signature() instead return _inspect.getargspec(target) /home/initial/miniconda3/envs/HOnnotate/lib/python3.5/site-packages/tensorflow/python/util/tf_inspect.py:75: DeprecationWarning: inspect.getargspec() is deprecated, use inspect.signature() instead return _inspect.getargspec(target) WARNING:tensorflow:From /home/initial/workspace/HOnnotate/optimization/ghope/loss.py:225: Print (from tensorflow.python.ops.logging_ops) is deprecated and will be removed after 2018-08-20. Instructions for updating: Use tf.print instead of tf.Print. Note that tf.print returns a no-output operator that directly prints the output. Outside of defuns or eager mode, this operator will not be executed unless it is directly specified in session.run or used as a control dependency for other operators. This is only a concern in graph mode. Below is an example of how to ensure tf.print executes in graph mode:
sess = tf.Session() with sess.as_default(): tensor = tf.range(10) print_op = tf.print(tensor) with tf.control_dependencies([print_op]): out = tf.add(tensor, tensor) sess.run(out) ``` Additionally, to use tf.print in python 2.7, users must make sure to import the following: `from __future__ import print_function` /home/initial/miniconda3/envs/HOnnotate/lib/python3.5/site-packages/chumpy/ch.py:1203: DeprecationWarning: inspect.getargspec() is deprecated, use inspect.signature() instead want_out = 'out' in inspect.getargspec(func).args /home/initial/miniconda3/envs/HOnnotate/lib/python3.5/site-packages/numpy/matrixlib/defmatrix.py:68: PendingDeprecationWarning: the matrix subclass is not the recommended way to represent matrices or deal with linear algebra (see https://docs.scipy.org/doc/numpy/user/numpy-for-matlab-users.html). Please adjust your code to use regular ndarray. return matrix(data, dtype=dtype, copy=False) /home/initial/workspace/HOnnotate/optimization/object/batch_object.py:65: RuntimeWarning: invalid value encountered in true_divide vn = mesh.vn / np.expand_dims(np.linalg.norm(mesh.vn, ord=2, axis=1), 1) # normalize to unit vec /home/initial/workspace/HOnnotate/optimization/ghope/loss.py:106: DeprecationWarning: Both axis > a.ndim and axis < -a.ndim - 1 are deprecated and will raise an AxisError in the future. gaussKernel = np.expand_dims(gaussKernel, 3) /home/initial/workspace/HOnnotate/optimization/ghope/loss.py:106: DeprecationWarning: Both axis > a.ndim and axis < -a.ndim - 1 are deprecated and will raise an AxisError in the future. gaussKernel = np.expand_dims(gaussKernel, 3) /home/initial/workspace/HOnnotate/optimization/ghope/loss.py:106: DeprecationWarning: Both axis > a.ndim and axis < -a.ndim - 1 are deprecated and will raise an AxisError in the future. gaussKernel = np.expand_dims(gaussKernel, 3) /home/initial/workspace/HOnnotate/optimization/ghope/loss.py:106: DeprecationWarning: Both axis > a.ndim and axis < -a.ndim - 1 are deprecated and will raise an AxisError in the future. gaussKernel = np.expand_dims(gaussKernel, 3) WARNING:tensorflow:From /home/initial/workspace/HOnnotate/optimization/ghope/icp.py:112: arg_min (from tensorflow.python.ops.gen_math_ops) is deprecated and will be removed in a future version. Instructions for updating: Use `argmin` instead 2024-01-14 10:50:00.623402: I tensorflow/core/platform/cpu_feature_guard.cc:141] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2 FMA 2024-01-14 10:50:00.999041: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1432] Found device 0 with properties: name: Quadro P5000 major: 6 minor: 1 memoryClockRate(GHz): 1.7335 pciBusID: 0000:03:00.0 totalMemory: 15.90GiB freeMemory: 15.78GiB 2024-01-14 10:50:01.314176: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1432] Found device 1 with properties: name: NVIDIA GeForce RTX 3090 major: 8 minor: 6 memoryClockRate(GHz): 1.725 pciBusID: 0000:02:00.0 totalMemory: 23.70GiB freeMemory: 22.76GiB 2024-01-14 10:50:01.314243: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1511] Adding visible gpu devices: 0, 1 2024-01-14 10:50:28.016592: I tensorflow/core/common_runtime/gpu/gpu_device.cc:982] Device interconnect StreamExecutor with strength 1 edge matrix: 2024-01-14 10:50:28.016640: I tensorflow/core/common_runtime/gpu/gpu_device.cc:988] 0 1 2024-01-14 10:50:28.016651: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1001] 0: N N 2024-01-14 10:50:28.016657: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1001] 1: N N 2024-01-14 10:50:28.016832: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1115] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 15293 MB memory) -> physical GPU (device: 0, name: Quadro P5000, pci bus id: 0000:03:00.0, compute capability: 6.1) 2024-01-14 10:50:28.033847: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1115] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:1 with 22064 MB memory) -> physical GPU (device: 1, name: NVIDIA GeForce RTX 3090, pci bus id: 0000:02:00.0, compute capability: 8.6) Clipping input data to the valid range for imshow with RGB data ([0..1] for floats or [0..255] for integers). maskPC for Image test/0/00002 is 0.132587 [Loading New frame ][0/00002] 2024-01-14 10:50:47.882958: W tensorflow/core/common_runtime/bfc_allocator.cc:267] Allocator (GPU_0_bfc) ran out of memory trying to allocate 29.75GiB. Current allocation summary follows. 2024-01-14 10:50:47.883086: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (256): Total Chunks: 68, Chunks in use: 66. 17.0KiB allocated for chunks. 16.5KiB in use in bin. 788B client-requested in use in bin. 2024-01-14 10:50:47.883121: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (512): Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin. 2024-01-14 10:50:47.883151: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (1024): Total Chunks: 1, Chunks in use: 1. 1.2KiB allocated for chunks. 1.2KiB in use in bin. 1.0KiB client-requested in use in bin. 2024-01-14 10:50:47.883177: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (2048): Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin. 2024-01-14 10:50:47.883202: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (4096): Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin. 2024-01-14 10:50:47.883229: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (8192): Total Chunks: 2, Chunks in use: 2. 18.5KiB allocated for chunks. 18.5KiB in use in bin. 18.2KiB client-requested in use in bin. 2024-01-14 10:50:47.883253: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (16384): Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin. 2024-01-14 10:50:47.883281: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (32768): Total Chunks: 2, Chunks in use: 2. 97.5KiB allocated for chunks. 97.5KiB in use in bin. 97.2KiB client-requested in use in bin. 2024-01-14 10:50:47.883309: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (65536): Total Chunks: 3, Chunks in use: 3. 301.8KiB allocated for chunks. 301.8KiB in use in bin. 301.4KiB client-requested in use in bin. 2024-01-14 10:50:47.883358: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (131072): Total Chunks: 1, Chunks in use: 0. 134.8KiB allocated for chunks. 0B in use in bin. 0B client-requested in use in bin. 2024-01-14 10:50:47.883390: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (262144): Total Chunks: 3, Chunks in use: 3. 1014.2KiB allocated for chunks. 1014.2KiB in use in bin. 900.0KiB client-requested in use in bin. 2024-01-14 10:50:47.883417: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (524288): Total Chunks: 10, Chunks in use: 10. 8.79MiB allocated for chunks. 8.79MiB in use in bin. 8.79MiB client-requested in use in bin. 2024-01-14 10:50:47.883442: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (1048576): Total Chunks: 4, Chunks in use: 3. 5.17MiB allocated for chunks. 4.00MiB in use in bin. 3.84MiB client-requested in use in bin. 2024-01-14 10:50:47.883468: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (2097152): Total Chunks: 2, Chunks in use: 2. 5.61MiB allocated for chunks. 5.61MiB in use in bin. 4.20MiB client-requested in use in bin. 2024-01-14 10:50:47.883495: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (4194304): Total Chunks: 5, Chunks in use: 3. 23.89MiB allocated for chunks. 14.00MiB in use in bin. 14.00MiB client-requested in use in bin. 2024-01-14 10:50:47.883525: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (8388608): Total Chunks: 2, Chunks in use: 2. 24.00MiB allocated for chunks. 24.00MiB in use in bin. 20.00MiB client-requested in use in bin. 2024-01-14 10:50:47.883550: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (16777216): Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin. 2024-01-14 10:50:47.883575: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (33554432): Total Chunks: 1, Chunks in use: 0. 58.00MiB allocated for chunks. 0B in use in bin. 0B client-requested in use in bin. 2024-01-14 10:50:47.883600: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (67108864): Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin. 2024-01-14 10:50:47.883625: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (134217728): Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin. 2024-01-14 10:50:47.883649: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (268435456): Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin. 2024-01-14 10:50:47.883675: I tensorflow/core/common_runtime/bfc_allocator.cc:613] Bin for 29.75GiB was 256.00MiB, Chunk State: 2024-01-14 10:50:47.883700: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7fc7d0000000 of size 6291712 2024-01-14 10:50:47.883722: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Free at 0x7fc7d0600100 of size 60817152 2024-01-14 10:50:47.883743: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7fc7e6000000 of size 12582912 2024-01-14 10:50:47.883763: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Free at 0x7fc7e6c00000 of size 4194560 2024-01-14 10:50:47.883784: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7fc7e7000100 of size 4194560 2024-01-14 10:50:47.883805: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7fc7e7400200 of size 12582400 2024-01-14 10:50:47.883825: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7fc7ea800000 of size 307200 2024-01-14 10:50:47.883845: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7fc7ea84b000 of size 256 2024-01-14 10:50:47.883865: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7fc7ea84b100 of size 4194560 2024-01-14 10:50:47.883885: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7fc7eac4b200 of size 256 2024-01-14 10:50:47.883905: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7fc7eac4b300 of size 256 2024-01-14 10:50:47.883924: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7fc7eac4b400 of size 256 2024-01-14 10:50:47.883944: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7fc7eac4b500 of size 256 2024-01-14 10:50:47.883963: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7fc7eac4b600 of size 256 2024-01-14 10:50:47.883982: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7fc7eac4b700 of size 256 2024-01-14 10:50:47.884001: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7fc7eac4b800 of size 256 2024-01-14 10:50:47.884021: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7fc7eac4b900 of size 256 2024-01-14 10:50:47.884040: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7fc7eac4ba00 of size 921600 2024-01-14 10:50:47.884060: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7fc7ead2ca00 of size 256 2024-01-14 10:50:47.884079: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7fc7ead2cb00 of size 256 2024-01-14 10:50:47.884098: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7fc7ead2cc00 of size 921600 2024-01-14 10:50:47.884117: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7fc7eae0dc00 of size 921600 2024-01-14 10:50:47.884136: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7fc7eaeeec00 of size 256 2024-01-14 10:50:47.884155: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7fc7eaeeed00 of size 256 2024-01-14 10:50:47.884174: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7fc7eaeeee00 of size 256 2024-01-14 10:50:47.884193: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7fc7eaeeef00 of size 256 2024-01-14 10:50:47.884213: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7fc7eaeef000 of size 256 2024-01-14 10:50:47.884233: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Free at 0x7fc7eaeef100 of size 256 2024-01-14 10:50:47.884251: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7fc7eaeef200 of size 256 2024-01-14 10:50:47.884270: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7fc7eaeef300 of size 256 2024-01-14 10:50:47.884289: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7fc7eaeef400 of size 256 2024-01-14 10:50:47.884308: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7fc7eaeef500 of size 256 2024-01-14 10:50:47.884327: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7fc7eaeef600 of size 256 2024-01-14 10:50:47.884346: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7fc7eaeef700 of size 256 2024-01-14 10:50:47.884365: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7fc7eaeef800 of size 256 2024-01-14 10:50:47.884384: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7fc7eaeef900 of size 256 2024-01-14 10:50:47.884403: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7fc7eaeefa00 of size 256 2024-01-14 10:50:47.884421: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Free at 0x7fc7eaeefb00 of size 256 2024-01-14 10:50:47.884440: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7fc7eaeefc00 of size 256 2024-01-14 10:50:47.884459: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7fc7eaeefd00 of size 921600 2024-01-14 10:50:47.884479: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Free at 0x7fc7eafd0d00 of size 1228800 2024-01-14 10:50:47.884498: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7fc7eb0fcd00 of size 921600 2024-01-14 10:50:47.884517: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Free at 0x7fc7eb1ddd00 of size 137984 2024-01-14 10:50:47.884536: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7fc7eb1ff800 of size 122112 2024-01-14 10:50:47.884556: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Free at 0x7fc7eb21d500 of size 6171392 2024-01-14 10:50:47.884576: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7fc8f2a00000 of size 9472 2024-01-14 10:50:47.884596: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7fc8f2a02500 of size 256 2024-01-14 10:50:47.884615: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7fc8f2a02600 of size 256 2024-01-14 10:50:47.884635: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7fc8f2a02700 of size 93440 2024-01-14 10:50:47.884654: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7fc8f2a19400 of size 256 2024-01-14 10:50:47.884674: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7fc8f2a19500 of size 49920 2024-01-14 10:50:47.884693: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7fc8f2a25800 of size 256 2024-01-14 10:50:47.884712: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7fc8f2a25900 of size 256 2024-01-14 10:50:47.884731: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7fc8f2a25a00 of size 256 2024-01-14 10:50:47.884751: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7fc8f2a25b00 of size 1280 2024-01-14 10:50:47.884771: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7fc8f2a26000 of size 9472 2024-01-14 10:50:47.884790: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7fc8f2a28500 of size 256 2024-01-14 10:50:47.884810: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7fc8f2a28600 of size 256 2024-01-14 10:50:47.884829: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7fc8f2a28700 of size 256 2024-01-14 10:50:47.884848: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7fc8f2a28800 of size 256 2024-01-14 10:50:47.884867: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7fc8f2a28900 of size 256 2024-01-14 10:50:47.884887: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7fc8f2a28a00 of size 256 2024-01-14 10:50:47.884906: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7fc8f2a28b00 of size 93440 2024-01-14 10:50:47.884925: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7fc8f2a3f800 of size 256 2024-01-14 10:50:47.884944: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7fc8f2a3f900 of size 49920 2024-01-14 10:50:47.884963: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7fc8f2a4bc00 of size 256 2024-01-14 10:50:47.884983: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7fc8f2a4bd00 of size 256 2024-01-14 10:50:47.885002: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7fc8f2a4be00 of size 256 2024-01-14 10:50:47.885021: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7fc8f2a4bf00 of size 256 2024-01-14 10:50:47.885040: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7fc8f2a4c000 of size 256 2024-01-14 10:50:47.885059: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7fc8f2a4c100 of size 256 2024-01-14 10:50:47.885078: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7fc8f2a4c200 of size 256 2024-01-14 10:50:47.885098: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7fc8f2a4c300 of size 256 2024-01-14 10:50:47.885118: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7fc8f2a4c400 of size 256 2024-01-14 10:50:47.885138: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7fc8f2a4c500 of size 256 2024-01-14 10:50:47.885157: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7fc8f2a4c600 of size 256 2024-01-14 10:50:47.885177: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7fc8f2a4c700 of size 256 2024-01-14 10:50:47.885196: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7fc8f2a4c800 of size 256 2024-01-14 10:50:47.885215: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7fc8f2a4c900 of size 256 2024-01-14 10:50:47.885235: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7fc8f2a4ca00 of size 256 2024-01-14 10:50:47.885253: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7fc8f2a4cb00 of size 256 2024-01-14 10:50:47.885272: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7fc8f2a4cc00 of size 256 2024-01-14 10:50:47.885292: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7fc8f2a4cd00 of size 256 2024-01-14 10:50:47.885311: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7fc8f2a4ce00 of size 256 2024-01-14 10:50:47.885330: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7fc8f2a4cf00 of size 256 2024-01-14 10:50:47.885349: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7fc8f2a4d000 of size 256 2024-01-14 10:50:47.885368: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7fc8f2a4d100 of size 256 2024-01-14 10:50:47.885387: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7fc8f2a4d200 of size 256 2024-01-14 10:50:47.885406: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7fc8f2a4d300 of size 256 2024-01-14 10:50:47.885424: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7fc8f2a4d400 of size 256 2024-01-14 10:50:47.885444: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7fc8f2a4d500 of size 256 2024-01-14 10:50:47.885464: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7fc8f2a4d600 of size 256 2024-01-14 10:50:47.885482: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7fc8f2a4d700 of size 307200 2024-01-14 10:50:47.885502: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7fc8f2a98700 of size 424192 2024-01-14 10:50:47.885523: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7fc8f2e00000 of size 2097152 2024-01-14 10:50:47.885543: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7fc8f3000000 of size 921600 2024-01-14 10:50:47.885563: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7fc8f30e1000 of size 1260544 2024-01-14 10:50:47.885582: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7fc8f3214c00 of size 921600 2024-01-14 10:50:47.885602: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7fc8f32f5c00 of size 1090560 2024-01-14 10:50:47.885622: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7fc8f3400000 of size 921600 2024-01-14 10:50:47.885640: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7fc8f34e1000 of size 921600 2024-01-14 10:50:47.885659: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7fc8f35c2000 of size 921600 2024-01-14 10:50:47.885679: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7fc8f36a3000 of size 1843200 2024-01-14 10:50:47.885700: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7fc8f3865000 of size 3780608 2024-01-14 10:50:47.885720: I tensorflow/core/common_runtime/bfc_allocator.cc:638] Summary of in-use Chunks by size: 2024-01-14 10:50:47.885745: I tensorflow/core/common_runtime/bfc_allocator.cc:641] 66 Chunks of size 256 totalling 16.5KiB 2024-01-14 10:50:47.885769: I tensorflow/core/common_runtime/bfc_allocator.cc:641] 1 Chunks of size 1280 totalling 1.2KiB 2024-01-14 10:50:47.885791: I tensorflow/core/common_runtime/bfc_allocator.cc:641] 2 Chunks of size 9472 totalling 18.5KiB 2024-01-14 10:50:47.885813: I tensorflow/core/common_runtime/bfc_allocator.cc:641] 2 Chunks of size 49920 totalling 97.5KiB 2024-01-14 10:50:47.885836: I tensorflow/core/common_runtime/bfc_allocator.cc:641] 2 Chunks of size 93440 totalling 182.5KiB 2024-01-14 10:50:47.885858: I tensorflow/core/common_runtime/bfc_allocator.cc:641] 1 Chunks of size 122112 totalling 119.2KiB 2024-01-14 10:50:47.885880: I tensorflow/core/common_runtime/bfc_allocator.cc:641] 2 Chunks of size 307200 totalling 600.0KiB 2024-01-14 10:50:47.885901: I tensorflow/core/common_runtime/bfc_allocator.cc:641] 1 Chunks of size 424192 totalling 414.2KiB 2024-01-14 10:50:47.885923: I tensorflow/core/common_runtime/bfc_allocator.cc:641] 10 Chunks of size 921600 totalling 8.79MiB 2024-01-14 10:50:47.885945: I tensorflow/core/common_runtime/bfc_allocator.cc:641] 1 Chunks of size 1090560 totalling 1.04MiB 2024-01-14 10:50:47.885966: I tensorflow/core/common_runtime/bfc_allocator.cc:641] 1 Chunks of size 1260544 totalling 1.20MiB 2024-01-14 10:50:47.885987: I tensorflow/core/common_runtime/bfc_allocator.cc:641] 1 Chunks of size 1843200 totalling 1.76MiB 2024-01-14 10:50:47.886008: I tensorflow/core/common_runtime/bfc_allocator.cc:641] 1 Chunks of size 2097152 totalling 2.00MiB 2024-01-14 10:50:47.886029: I tensorflow/core/common_runtime/bfc_allocator.cc:641] 1 Chunks of size 3780608 totalling 3.61MiB 2024-01-14 10:50:47.886050: I tensorflow/core/common_runtime/bfc_allocator.cc:641] 2 Chunks of size 4194560 totalling 8.00MiB 2024-01-14 10:50:47.886071: I tensorflow/core/common_runtime/bfc_allocator.cc:641] 1 Chunks of size 6291712 totalling 6.00MiB 2024-01-14 10:50:47.886094: I tensorflow/core/common_runtime/bfc_allocator.cc:641] 1 Chunks of size 12582400 totalling 12.00MiB 2024-01-14 10:50:47.886116: I tensorflow/core/common_runtime/bfc_allocator.cc:641] 1 Chunks of size 12582912 totalling 12.00MiB 2024-01-14 10:50:47.886138: I tensorflow/core/common_runtime/bfc_allocator.cc:645] Sum Total of in-use chunks: 57.81MiB 2024-01-14 10:50:47.886165: I tensorflow/core/common_runtime/bfc_allocator.cc:647] Stats: Limit: 16035977626 InUse: 60618752 MaxInUse: 64813312 NumAllocs: 139 MaxAllocSize: 12582912 2024-01-14 10:50:47.886214: W tensorflow/core/common_runtime/bfc_allocator.cc:271] *****_____________________________________________**********__***********xx*********____************ 2024-01-14 10:50:47.901539: W tensorflow/core/framework/op_kernel.cc:1273] OP_REQUIRES failed at cwise_ops_common.cc:70 : Resource exhausted: OOM when allocating tensor with shape[7617,262146,4] and type float on /job:localhost/replica:0/task:0/device:GPU:0 by allocator GPU_0_bfc 2024-01-14 10:50:48.043748: I /home/initial/workspace/thirdparty_for_HOnnotate/dirt/csrc/gl_common.h:66] selected egl device #1 to match cuda device #0 for thread 0x7fc7e5fff700 2024-01-14 10:50:48.303976: I /home/initial/workspace/thirdparty_for_HOnnotate/dirt/csrc/gl_common.h:84] successfully created new GL context on thread 0x7fc7e5fff700 (EGL = 1.5, GL = 4.6.0 NVIDIA 470.223.02, renderer = Quadro P5000/PCIe/SSE2) 2024-01-14 10:50:48.664465: I /home/initial/workspace/thirdparty_for_HOnnotate/dirt/csrc/rasterise_egl.cpp:266] reinitialised framebuffer with size 320 x 480 2024-01-14 10:50:48.726655: I /home/initial/workspace/thirdparty_for_HOnnotate/dirt/csrc/gl_common.h:66] selected egl device #1 to match cuda device #0 for thread 0x7fc7e57fe700 2024-01-14 10:50:48.749039: I /home/initial/workspace/thirdparty_for_HOnnotate/dirt/csrc/gl_common.h:84] successfully created new GL context on thread 0x7fc7e57fe700 (EGL = 1.5, GL = 4.6.0 NVIDIA 470.223.02, renderer = Quadro P5000/PCIe/SSE2) 2024-01-14 10:50:48.751036: I /home/initial/workspace/thirdparty_for_HOnnotate/dirt/csrc/rasterise_grad_egl.cpp:314] reinitialised framebuffer with size 320 x 480 Traceback (most recent call last): File "/home/initial/miniconda3/envs/HOnnotate/lib/python3.5/site-packages/tensorflow/python/client/session.py", line 1334, in _do_call return fn(*args) File "/home/initial/miniconda3/envs/HOnnotate/lib/python3.5/site-packages/tensorflow/python/client/session.py", line 1319, in _run_fn options, feed_dict, fetch_list, target_list, run_metadata) File "/home/initial/miniconda3/envs/HOnnotate/lib/python3.5/site-packages/tensorflow/python/client/session.py", line 1407, in _call_tf_sessionrun run_metadata) tensorflow.python.framework.errors_impl.ResourceExhaustedError: OOM when allocating tensor with shape[7617,262146,4] and type float on /job:localhost/replica:0/task:0/device:GPU:0 by allocator GPU_0_bfc [[{{node map/while/sub_1}} = Sub[T=DT_FLOAT, _device="/job:localhost/replica:0/task:0/device:GPU:0"](map/while/ExpandDims, map/while/TensorArrayReadV3)]] Hint: If you want to see a list of allocated tensors when OOM happens, add report_tensor_allocations_upon_oom to RunOptions for current allocation info. During handling of the above exception, another exception occurred: Traceback (most recent call last): File "objectTrackingSingleFrame.py", line 358, in <module> objectTracker(w, h, paramInit, camProp, mesh, out_dir, configData) File "objectTrackingSingleFrame.py", line 156, in objectTracker opti1.runOptimization(session, 1, {loadData:True}) File "/home/initial/workspace/HOnnotate/optimization/ghope/utils.py", line 159, in timed result = method(*args, **kw) File "/home/initial/workspace/HOnnotate/optimization/ghope/optimization.py", line 68, in runOptimization session.run(self.optOp, feed_dict=feedDict) File "/home/initial/miniconda3/envs/HOnnotate/lib/python3.5/site-packages/tensorflow/python/client/session.py", line 929, in run run_metadata_ptr) File "/home/initial/miniconda3/envs/HOnnotate/lib/python3.5/site-packages/tensorflow/python/client/session.py", line 1152, in _run feed_dict_tensor, options, run_metadata) File "/home/initial/miniconda3/envs/HOnnotate/lib/python3.5/site-packages/tensorflow/python/client/session.py", line 1328, in _do_run run_metadata) File "/home/initial/miniconda3/envs/HOnnotate/lib/python3.5/site-packages/tensorflow/python/client/session.py", line 1348, in _do_call raise type(e)(node_def, op, message) tensorflow.python.framework.errors_impl.ResourceExhaustedError: OOM when allocating tensor with shape[7617,262146,4] and type float on /job:localhost/replica:0/task:0/device:GPU:0 by allocator GPU_0_bfc [[node map/while/sub_1 (defined at /home/initial/workspace/HOnnotate/optimization/ghope/icp.py:112) = Sub[T=DT_FLOAT, _device="/job:localhost/replica:0/task:0/device:GPU:0"](map/while/ExpandDims, map/while/TensorArrayReadV3)]] Hint: If you want to see a list of allocated tensors when OOM happens, add report_tensor_allocations_upon_oom to RunOptions for current allocation info. Caused by op 'map/while/sub_1', defined at: File "objectTrackingSingleFrame.py", line 358, in <module> objectTracker(w, h, paramInit, camProp, mesh, out_dir, configData) File "objectTrackingSingleFrame.py", line 92, in objectTracker icpLoss = icp.getLoss(finalMesh.vUnClipped) File "/home/initial/workspace/HOnnotate/optimization/ghope/icp.py", line 191, in getLoss loss = tf.map_fn(self.getLossPerFrame, (finalMesh, self.pcl2DAllViews, self.realObservs.seg, segColor), parallel_iterations=int(self.numViews), dtype=tf.float32) File "/home/initial/miniconda3/envs/HOnnotate/lib/python3.5/site-packages/tensorflow/python/ops/functional_ops.py", line 494, in map_fn maximum_iterations=n) File "/home/initial/miniconda3/envs/HOnnotate/lib/python3.5/site-packages/tensorflow/python/ops/control_flow_ops.py", line 3291, in while_loop return_same_structure) File "/home/initial/miniconda3/envs/HOnnotate/lib/python3.5/site-packages/tensorflow/python/ops/control_flow_ops.py", line 3004, in BuildLoop pred, body, original_loop_vars, loop_vars, shape_invariants) File "/home/initial/miniconda3/envs/HOnnotate/lib/python3.5/site-packages/tensorflow/python/ops/control_flow_ops.py", line 2939, in _BuildLoop body_result = body(*packed_vars_for_body) File "/home/initial/miniconda3/envs/HOnnotate/lib/python3.5/site-packages/tensorflow/python/ops/control_flow_ops.py", line 3260, in <lambda> body = lambda i, lv: (i + 1, orig_body(*lv)) File "/home/initial/miniconda3/envs/HOnnotate/lib/python3.5/site-packages/tensorflow/python/ops/functional_ops.py", line 483, in compute packed_fn_values = fn(packed_values) File "/home/initial/workspace/HOnnotate/optimization/ghope/icp.py", line 112, in getLossPerFrame tf.arg_min(tf.reduce_sum(tf.abs(pclValid - verts), axis=2), dimension=1)) # N, File "/home/initial/miniconda3/envs/HOnnotate/lib/python3.5/site-packages/tensorflow/python/ops/math_ops.py", line 866, in binary_op_wrapper return func(x, y, name=name) File "/home/initial/miniconda3/envs/HOnnotate/lib/python3.5/site-packages/tensorflow/python/ops/gen_math_ops.py", line 8318, in sub "Sub", x=x, y=y, name=name) File "/home/initial/miniconda3/envs/HOnnotate/lib/python3.5/site-packages/tensorflow/python/framework/op_def_library.py", line 787, in _apply_op_helper op_def=op_def) File "/home/initial/miniconda3/envs/HOnnotate/lib/python3.5/site-packages/tensorflow/python/util/deprecation.py", line 488, in new_func return func(*args, **kwargs) File "/home/initial/miniconda3/envs/HOnnotate/lib/python3.5/site-packages/tensorflow/python/framework/ops.py", line 3274, in create_op op_def=op_def) File "/home/initial/miniconda3/envs/HOnnotate/lib/python3.5/site-packages/tensorflow/python/framework/ops.py", line 1770, in __init__ self._traceback = tf_stack.extract_stack() ResourceExhaustedError (see above for traceback): OOM when allocating tensor with shape[7617,262146,4] and type float on /job:localhost/replica:0/task:0/device:GPU:0 by allocator GPU_0_bfc [[node map/while/sub_1 (defined at /home/initial/workspace/HOnnotate/optimization/ghope/icp.py:112) = Sub[T=DT_FLOAT, _device="/job:localhost/replica:0/task:0/device:GPU:0"](map/while/ExpandDims, map/while/TensorArrayReadV3)]] Hint: If you want to see a list of allocated tensors when OOM happens, add report_tensor_allocations_upon_oom to RunOptions for current allocation info.
Hello brother, I have the same problem as you. Have you solved it now?
from honnotate.
Related Issues (20)
- about ho3d_v2 competition in codelab HOT 3
- Estimate hand-object 3D pose from a first-person perspective
- ho3d_v2 challenge in codelab is down
- What's the difference between 'objCorners3D' and 'objCorners3DRest'? HOT 1
- What is the frame rate of the sequence? HOT 3
- The CodaLab Challenge was down, could you repair it. HOT 1
- The F-scores results of the HO3D-v2 Codalab Challenge are all zeros HOT 5
- Can/How I evaluate only the 21 hand 3d joints on HO3D? HOT 2
- Docker Image HOT 1
- does this code support tensorflow 2.x? HOT 1
- HO3D_v2 dataset evaluation HOT 1
- ho3d_v2 competition in codelab:a decision is pending HOT 2
- 3d keypoints of Ho3d
- prediction error HOT 1
- The units in the camera coordinate system?
- Can't download ho3d dataset-version 2
- HO3D_v2 challenge in codalab is down HOT 6
- Execution time limit exceeded for HO3D_v3 challenge
- Decision is pending. HOT 3
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from honnotate.