python prepare_egohands.py about hand-detection-tutorial HOT 21 CLOSED

The server hosting the "egohands" dataset seems to refuse connections now. You might try it later. Otherwise, you might need to contact vision.soic.indiana.edu for access of the file...

$ wget http://vision.soic.indiana.edu/egohands_files/egohands_data.zip
--2020-03-10 16:21:02--  http://vision.soic.indiana.edu/egohands_files/egohands_data.zip
Resolving vision.soic.indiana.edu (vision.soic.indiana.edu)... 156.56.83.21
Connecting to vision.soic.indiana.edu (vision.soic.indiana.edu)|156.56.83.21|:80... failed: Connection refused.

from hand-detection-tutorial.

So I can't apply that tutorial anymore?

from hand-detection-tutorial.

I don't know whether the dataset server is just temporarily out of service, or moved to another location, or else.

You might try to download it later, or try to find other sources of the same dataset.

from hand-detection-tutorial.

Thank you for your answers.

I tried from the beginning to try your tf_trt_models. Because I think doing it from the beginning would be easier to understand.

In fact, I have completed my training with my dataset.
I have frozen_inference_graph.pb, model.ckpt.data-00000-of-00001, model.ckpt.index, model.ckpt.neta, pipeline.config, modified ssd_mobilenet_v2_coco.config file etc.

I want to run tf_trt_models as a model I trained. Can you tell me where to put the file?

from hand-detection-tutorial.

Please refer to my jkjung-avt/tf_trt_models/README.md.

Be sure to copy your trained model checkpoint files into the corresponding data/xxx_egohands/ folder.

from hand-detection-tutorial.

I copied
'checkpoint, frozen_inference_graph.pb, model.ckpt.data-00000-of-00001, model.ckpt.index, model.ckpt.meta, model.ckpt-100000.data-00000-of-00001, model.ckpt-10000.index, model.ckpt-100000.meta, object-detection.pbtxt, pipeline.config, modified ssd_mobilenet_v2_coco.config'
into the ssd_mobilenet_v2_egohands file.

And these commands were entered.
python3 camera_tf_trt.py --image --filename image1.jpg --model ssd_mobilenet_v2_egohands --labelmap data/object-detection.pbtxt --num-classes 1 --build

Then this error occurred.
NotFoundError (see above for traceback): Key FeatureExtractor/MobilenetV2/layer_19_2_Conv2d_2_3x3_s2_512_depthwise/BatchNorm/beta not found in checkpoint [[Node: save/RestoreV2 = RestoreV2[dtypes=[DT_FLOAT, DT_FLOAT, DT_FLOAT, DT_FLOAT, DT_FLOAT, ..., DT_FLOAT, DT_FLOAT, DT_FLOAT, DT_FLOAT, DT_INT64], _device="/job:localhost/replica:0/task:0/device:CPU:0"](_arg_save/Const_0_0, save/RestoreV2/tensor_names, save/RestoreV2/shape_and_slices)]]

How to solve it?

from hand-detection-tutorial.

The overall results of implementation are like this.

/home/hyebin/anaconda3/lib/python3.6/site-packages/tensorflow/python/framework/dtypes.py:519: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
_np_qint8 = np.dtype([("qint8", np.int8, 1)])
/home/hyebin/anaconda3/lib/python3.6/site-packages/tensorflow/python/framework/dtypes.py:520: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
_np_quint8 = np.dtype([("quint8", np.uint8, 1)])
/home/hyebin/anaconda3/lib/python3.6/site-packages/tensorflow/python/framework/dtypes.py:521: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
_np_qint16 = np.dtype([("qint16", np.int16, 1)])
/home/hyebin/anaconda3/lib/python3.6/site-packages/tensorflow/python/framework/dtypes.py:522: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
_np_quint16 = np.dtype([("quint16", np.uint16, 1)])
/home/hyebin/anaconda3/lib/python3.6/site-packages/tensorflow/python/framework/dtypes.py:523: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
_np_qint32 = np.dtype([("qint32", np.int32, 1)])
/home/hyebin/anaconda3/lib/python3.6/site-packages/tensorflow/python/framework/dtypes.py:528: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
np_resource = np.dtype([("resource", np.ubyte, 1)])
INFO:main:called with args: Namespace(conf_th=0.3, do_build=True, do_tensorboard=False, filename='image1.jpg', image_height=480, image_width=640, labelmap_file='data/object-detection.pbtxt', model='ssd_mobilenet_v2_egohands', num_classes=1, rtsp_latency=200, rtsp_uri=None, use_file=False, use_image=True, use_rtsp=False, use_usb=False, video_dev=0)
INFO:main:reading label map
INFO:main:building TRT graph and saving to pb: ./data/ssd_mobilenet_v2_egohands_trt.pb
2020-03-11 15:33:12.228887: I tensorflow/core/platform/cpu_feature_guard.cc:140] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2 FMA
INFO:tensorflow:depth of additional conv before box predictor: 0
INFO:tensorflow:depth of additional conv before box predictor: 0
INFO:tensorflow:depth of additional conv before box predictor: 0
INFO:tensorflow:depth of additional conv before box predictor: 0
INFO:tensorflow:depth of additional conv before box predictor: 0
INFO:tensorflow:depth of additional conv before box predictor: 0
WARNING:tensorflow:From /home/hyebin/.local/lib/python3.6/site-packages/object_detection-0.1-py3.6.egg/object_detection/exporter.py:356: get_or_create_global_step (from tensorflow.contrib.framework.python.ops.variables) is deprecated and will be removed in a future version.
Instructions for updating:
Please switch to tf.train.get_or_create_global_step
INFO:tensorflow:Restoring parameters from data/ssd_mobilenet_v2_egohands/model.ckpt-10000
2020-03-11 15:33:15.034457: W tensorflow/core/framework/op_kernel.cc:1318] OP_REQUIRES failed at save_restore_v2_ops.cc:184 : Not found: Key FeatureExtractor/MobilenetV2/layer_19_2_Conv2d_2_3x3_s2_512_depthwise/BatchNorm/beta not found in checkpoint
Traceback (most recent call last):
File "/home/hyebin/anaconda3/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1322, in _do_call
return fn(*args)
File "/home/hyebin/anaconda3/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1307, in _run_fn
options, feed_dict, fetch_list, target_list, run_metadata)
File "/home/hyebin/anaconda3/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1409, in _call_tf_sessionrun
run_metadata)
tensorflow.python.framework.errors_impl.NotFoundError: Key FeatureExtractor/MobilenetV2/layer_19_2_Conv2d_2_3x3_s2_512_depthwise/BatchNorm/beta not found in checkpoint
[[Node: save/RestoreV2 = RestoreV2[dtypes=[DT_FLOAT, DT_FLOAT, DT_FLOAT, DT_FLOAT, DT_FLOAT, ..., DT_FLOAT, DT_FLOAT, DT_FLOAT, DT_FLOAT, DT_INT64], _device="/job:localhost/replica:0/task:0/device:CPU:0"](_arg_save/Const_0_0, save/RestoreV2/tensor_names, save/RestoreV2/shape_and_slices)]]

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "camera_tf_trt.py", line 199, in
main()
File "camera_tf_trt.py", line 158, in main
build_trt_pb(args.model, pb_path)
File "/home/hyebin/tf_trt_models/utils/od_utils.py", line 55, in build_trt_pb
checkpoint=checkpoint_path
File "/home/hyebin/tf_trt_models/tf_trt_models/detection.py", line 165, in build_detection_graph
input_shape=[batch_size, None, None, 3]
File "/home/hyebin/.local/lib/python3.6/site-packages/object_detection-0.1-py3.6.egg/object_detection/exporter.py", line 474, in export_inference_graph
write_inference_graph=write_inference_graph)
File "/home/hyebin/.local/lib/python3.6/site-packages/object_detection-0.1-py3.6.egg/object_detection/exporter.py", line 409, in _export_inference_graph
trained_checkpoint_prefix=checkpoint_to_use)
File "/home/hyebin/.local/lib/python3.6/site-packages/object_detection-0.1-py3.6.egg/object_detection/exporter.py", line 321, in write_graph_and_checkpoint
saver.restore(sess, trained_checkpoint_prefix)
File "/home/hyebin/anaconda3/lib/python3.6/site-packages/tensorflow/python/training/saver.py", line 1802, in restore
{self.saver_def.filename_tensor_name: save_path})
File "/home/hyebin/anaconda3/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 900, in run
run_metadata_ptr)
File "/home/hyebin/anaconda3/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1135, in _run
feed_dict_tensor, options, run_metadata)
File "/home/hyebin/anaconda3/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1316, in _do_run
run_metadata)
File "/home/hyebin/anaconda3/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1335, in _do_call
raise type(e)(node_def, op, message)
tensorflow.python.framework.errors_impl.NotFoundError: Key FeatureExtractor/MobilenetV2/layer_19_2_Conv2d_2_3x3_s2_512_depthwise/BatchNorm/beta not found in checkpoint
[[Node: save/RestoreV2 = RestoreV2[dtypes=[DT_FLOAT, DT_FLOAT, DT_FLOAT, DT_FLOAT, DT_FLOAT, ..., DT_FLOAT, DT_FLOAT, DT_FLOAT, DT_FLOAT, DT_INT64], _device="/job:localhost/replica:0/task:0/device:CPU:0"](_arg_save/Const_0_0, save/RestoreV2/tensor_names, save/RestoreV2/shape_and_slices)]]

Caused by op 'save/RestoreV2', defined at:
File "camera_tf_trt.py", line 199, in
main()
File "camera_tf_trt.py", line 158, in main
build_trt_pb(args.model, pb_path)
File "/home/hyebin/tf_trt_models/utils/od_utils.py", line 55, in build_trt_pb
checkpoint=checkpoint_path
File "/home/hyebin/tf_trt_models/tf_trt_models/detection.py", line 165, in build_detection_graph
input_shape=[batch_size, None, None, 3]
File "/home/hyebin/.local/lib/python3.6/site-packages/object_detection-0.1-py3.6.egg/object_detection/exporter.py", line 474, in export_inference_graph
write_inference_graph=write_inference_graph)
File "/home/hyebin/.local/lib/python3.6/site-packages/object_detection-0.1-py3.6.egg/object_detection/exporter.py", line 409, in _export_inference_graph
trained_checkpoint_prefix=checkpoint_to_use)
File "/home/hyebin/.local/lib/python3.6/site-packages/object_detection-0.1-py3.6.egg/object_detection/exporter.py", line 317, in write_graph_and_checkpoint
tf.import_graph_def(inference_graph_def, name='')
File "/home/hyebin/anaconda3/lib/python3.6/site-packages/tensorflow/python/util/deprecation.py", line 432, in new_func
return func(*args, **kwargs)
File "/home/hyebin/anaconda3/lib/python3.6/site-packages/tensorflow/python/framework/importer.py", line 513, in import_graph_def
_ProcessNewOps(graph)
File "/home/hyebin/anaconda3/lib/python3.6/site-packages/tensorflow/python/framework/importer.py", line 303, in _ProcessNewOps
for new_op in graph._add_new_tf_operations(compute_devices=False): # pylint: disable=protected-access
File "/home/hyebin/anaconda3/lib/python3.6/site-packages/tensorflow/python/framework/ops.py", line 3540, in _add_new_tf_operations
for c_op in c_api_util.new_tf_operations(self)
File "/home/hyebin/anaconda3/lib/python3.6/site-packages/tensorflow/python/framework/ops.py", line 3540, in
for c_op in c_api_util.new_tf_operations(self)
File "/home/hyebin/anaconda3/lib/python3.6/site-packages/tensorflow/python/framework/ops.py", line 3428, in _create_op_from_tf_operation
ret = Operation(c_op, self)
File "/home/hyebin/anaconda3/lib/python3.6/site-packages/tensorflow/python/framework/ops.py", line 1718, in init
self._traceback = self._graph._extract_stack() # pylint: disable=protected-access

NotFoundError (see above for traceback): Key FeatureExtractor/MobilenetV2/layer_19_2_Conv2d_2_3x3_s2_512_depthwise/BatchNorm/beta not found in checkpoint
[[Node: save/RestoreV2 = RestoreV2[dtypes=[DT_FLOAT, DT_FLOAT, DT_FLOAT, DT_FLOAT, DT_FLOAT, ..., DT_FLOAT, DT_FLOAT, DT_FLOAT, DT_FLOAT, DT_INT64], _device="/job:localhost/replica:0/task:0/device:CPU:0"](_arg_save/Const_0_0, save/RestoreV2/tensor_names, save/RestoreV2/shape_and_slices)]]

from hand-detection-tutorial.

~/tf_trt_models$ ls
build install.sh test.record
camera_tf_trt.py LICENSE.md tf_trt_models
data README.md tf_trt_models.egg-info
dist scripts third_party
examples setup.py train_labels.csv
face_inference_graph ssd_mobilenet_v2_coco.config train.record
image1.jpg test_labels.csv utils

~/tf_trt_models/data$ ls
classification_graphic.jpg rfcn_resnet101_egohands
detection_graphic.jpg rfcn_resnet101_egohands.config
egohands_label_map.pbtxt ssd_inception_v2_egohands
faster_rcnn_inception_v2_egohands ssd_inception_v2_egohands.config
faster_rcnn_inception_v2_egohands.config ssdlite_mobilenet_v2_egohands
faster_rcnn_resnet101_egohands ssdlite_mobilenet_v2_egohands.config
faster_rcnn_resnet101_egohands.config ssd_mobilenet_v1_coco_2018_01_28
faster_rcnn_resnet50_egohands ssd_mobilenet_v1_egohands
faster_rcnn_resnet50_egohands.config ssd_mobilenet_v1_egohands.config
huskies_detected.png ssd_mobilenet_v2_egohands
landing_graphic.jpg ssd_mobilenet_v2_egohands.config
object-detection.pbtxt ssd_mobilenet_v2_face.config
protoc

~/tf_trt_models/data/ssd_mobilenet_v2_egohands$ ls
checkpoint model.ckpt.meta
frozen_inference_graph.pb object-detection.pbtxt
model.ckpt-10000.data-00000-of-00001 pipeline.config
model.ckpt-10000.index README.md
model.ckpt-10000.meta saved_model
model.ckpt.data-00000-of-00001 ssd_mobilenet_v2_coco.config
model.ckpt.index

from hand-detection-tutorial.

Which version of tensorflow are you using on your Jetson TX2 or Nano?

Refer to Setup. I myself have tested 1.8.0 and 1.12.2.

from hand-detection-tutorial.

My jetsonTX2's tensorflow version is 2.0.

Doesn't it run on a desktop computer instead of a jetson?
My computer's tensorflow version is 1.8.

from hand-detection-tutorial.

No, I have not tested my tf_trt_models code on a x86_64 PC. It would likely require some modifications to work properly on your desktop PC. (In addition, you need to make sure your tensorflow binary supports TF-TRT. It would not work if you just pip installed tensorflow.)

On the other hand, if you are going to run tf_trt_models code on Jetson TX2, then you need to downgrade tensorflow to 1.x.

from hand-detection-tutorial.

I'll try it at Jetson. Thank you very much.

from hand-detection-tutorial.

I trited tf_trt_model in jetson tx2. Tensorflow version is 1.14.0.

This error occurred when it was implemented.

Original stack trace for 'save/RestoreV2': File "camera_tf_trt.py", line 199, in <module> main() File "camera_tf_trt.py", line 158, in main build_trt_pb(args.model, pb_path) File "/home/hyebin/tf_trt_models/utils/od_utils.py", line 55, in build_trt_pb checkpoint=checkpoint_path File "/home/hyebin/tf_trt_models/tf_trt_models/detection.py", line 165, in build_detection_graph input_shape=[batch_size, None, None, 3] File "/home/hyebin/.local/lib/python3.6/site-packages/object_detection-0.1-py3.6.egg/object_detection/exporter.py", line 474, in export_inference_graph write_inference_graph=write_inference_graph) File "/home/hyebin/.local/lib/python3.6/site-packages/object_detection-0.1-py3.6.egg/object_detection/exporter.py", line 409, in _export_inference_graph trained_checkpoint_prefix=checkpoint_to_use) File "/home/hyebin/.local/lib/python3.6/site-packages/object_detection-0.1-py3.6.egg/object_detection/exporter.py", line 317, in write_graph_and_checkpoint tf.import_graph_def(inference_graph_def, name='') File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/util/deprecation.py", line 507, in new_func return func(*args, **kwargs) File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/framework/importer.py", line 443, in import_graph_def _ProcessNewOps(graph) File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/framework/importer.py", line 236, in _ProcessNewOps for new_op in graph._add_new_tf_operations(compute_devices=False): # pylint: disable=protected-access File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/framework/ops.py", line 3751, in _add_new_tf_operations for c_op in c_api_util.new_tf_operations(self) File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/framework/ops.py", line 3751, in <listcomp> for c_op in c_api_util.new_tf_operations(self) File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/framework/ops.py", line 3641, in _create_op_from_tf_operation ret = Operation(c_op, self) File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/framework/ops.py", line 2005, in __init__ self._traceback = tf_stack.extract_stack()

from hand-detection-tutorial.

I don't know what to fix.

from hand-detection-tutorial.

All the results are like this.

hyebin@hyebin-desktop:~/tf_trt_models$ python3 camera_tf_trt.py --image --filename image1.jpeg --model ssd_mobilenet_v2_egohands --labelmap data/object-detection.pbtxt --num-classes 1 --build
2020-03-23 20:53:29.570514: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcudart.so.10.0
2020-03-23 20:53:41.847690: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcudart.so.10.0
INFO:main:called with args: Namespace(conf_th=0.3, do_build=True, do_tensorboard=False, filename='image1.jpeg', image_height=480, image_width=640, labelmap_file='data/object-detection.pbtxt', model='ssd_mobilenet_v2_egohands', num_classes=1, rtsp_latency=200, rtsp_uri=None, use_file=False, use_image=True, use_rtsp=False, use_usb=False, video_dev=0)
INFO:main:reading label map
WARNING:tensorflow:From /home/hyebin/.local/lib/python3.6/site-packages/object_detection-0.1-py3.6.egg/object_detection/utils/label_map_util.py:131: The name tf.gfile.GFile is deprecated. Please use tf.io.gfile.GFile instead.

INFO:main:building TRT graph and saving to pb: ./data/ssd_mobilenet_v2_egohands_trt.pb
WARNING:tensorflow:From /home/hyebin/.local/lib/python3.6/site-packages/slim-0.1-py3.6.egg/nets/inception_resnet_v2.py:373: The name tf.GraphKeys is deprecated. Please use tf.compat.v1.GraphKeys instead.

WARNING:tensorflow:From /home/hyebin/.local/lib/python3.6/site-packages/slim-0.1-py3.6.egg/nets/mobilenet/mobilenet.py:389: The name tf.nn.avg_pool is deprecated. Please use tf.nn.avg_pool2d instead.

WARNING:tensorflow:From /home/hyebin/tf_trt_models/tf_trt_models/detection.py:154: The name tf.ConfigProto is deprecated. Please use tf.compat.v1.ConfigProto instead.

WARNING:tensorflow:From /home/hyebin/tf_trt_models/tf_trt_models/detection.py:158: The name tf.Session is deprecated. Please use tf.compat.v1.Session instead.

2020-03-23 20:53:43.650015: W tensorflow/core/platform/profile_utils/cpu_utils.cc:98] Failed to find bogomips in /proc/cpuinfo; cannot determine CPU frequency
2020-03-23 20:53:43.651568: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x73a80b0 executing computations on platform Host. Devices:
2020-03-23 20:53:43.651698: I tensorflow/compiler/xla/service/service.cc:175] StreamExecutor device (0): ,
2020-03-23 20:53:43.664226: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcuda.so.1
2020-03-23 20:53:43.777658: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:972] ARM64 does not support NUMA - returning NUMA node zero
2020-03-23 20:53:43.778287: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x73e5710 executing computations on platform CUDA. Devices:
2020-03-23 20:53:43.778369: I tensorflow/compiler/xla/service/service.cc:175] StreamExecutor device (0): NVIDIA Tegra X2, Compute Capability 6.2
2020-03-23 20:53:43.779397: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:972] ARM64 does not support NUMA - returning NUMA node zero
2020-03-23 20:53:43.780231: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1640] Found device 0 with properties:
name: NVIDIA Tegra X2 major: 6 minor: 2 memoryClockRate(GHz): 1.02
pciBusID: 0000:00:00.0
2020-03-23 20:53:43.780909: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcudart.so.10.0
2020-03-23 20:53:43.781626: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcublas.so.10.0
2020-03-23 20:53:43.781810: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcufft.so.10.0
2020-03-23 20:53:43.781966: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcurand.so.10.0
2020-03-23 20:53:43.787734: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcusolver.so.10.0
2020-03-23 20:53:43.792670: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcusparse.so.10.0
2020-03-23 20:53:43.793004: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcudnn.so.7
2020-03-23 20:53:43.793498: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:972] ARM64 does not support NUMA - returning NUMA node zero
2020-03-23 20:53:43.794266: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:972] ARM64 does not support NUMA - returning NUMA node zero
2020-03-23 20:53:43.794510: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1763] Adding visible gpu devices: 0
2020-03-23 20:53:45.975451: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1181] Device interconnect StreamExecutor with strength 1 edge matrix:
2020-03-23 20:53:45.975591: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1187] 0
2020-03-23 20:53:45.975648: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1200] 0: N
2020-03-23 20:53:45.976317: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:972] ARM64 does not support NUMA - returning NUMA node zero
2020-03-23 20:53:45.976890: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:972] ARM64 does not support NUMA - returning NUMA node zero
2020-03-23 20:53:45.977341: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1326] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 269 MB memory) -> physical GPU (device: 0, name: NVIDIA Tegra X2, pci bus id: 0000:00:00.0, compute capability: 6.2)
WARNING:tensorflow:From /home/hyebin/.local/lib/python3.6/site-packages/object_detection-0.1-py3.6.egg/object_detection/exporter.py:374: The name tf.gfile.MakeDirs is deprecated. Please use tf.io.gfile.makedirs instead.

WARNING:tensorflow:From /home/hyebin/.local/lib/python3.6/site-packages/object_detection-0.1-py3.6.egg/object_detection/exporter.py:130: The name tf.placeholder is deprecated. Please use tf.compat.v1.placeholder instead.

WARNING:tensorflow:From /home/hyebin/.local/lib/python3.6/site-packages/object_detection-0.1-py3.6.egg/object_detection/exporter.py:327: to_float (from tensorflow.python.ops.math_ops) is deprecated and will be removed in a future version.
Instructions for updating:
Use tf.cast instead.
WARNING:tensorflow:From /home/hyebin/.local/lib/python3.6/site-packages/object_detection-0.1-py3.6.egg/object_detection/core/preprocessor.py:2340: The name tf.image.resize_images is deprecated. Please use tf.image.resize instead.

WARNING:tensorflow:From /home/hyebin/.local/lib/python3.6/site-packages/object_detection-0.1-py3.6.egg/object_detection/meta_architectures/ssd_meta_arch.py:390: The name tf.variable_scope is deprecated. Please use tf.compat.v1.variable_scope instead.

WARNING:tensorflow:From /home/hyebin/.local/lib/python3.6/site-packages/object_detection-0.1-py3.6.egg/object_detection/core/box_predictor.py:704: The name tf.logging.info is deprecated. Please use tf.compat.v1.logging.info instead.

INFO:tensorflow:depth of additional conv before box predictor: 0
INFO:tensorflow:depth of additional conv before box predictor: 0
INFO:tensorflow:depth of additional conv before box predictor: 0
INFO:tensorflow:depth of additional conv before box predictor: 0
INFO:tensorflow:depth of additional conv before box predictor: 0
INFO:tensorflow:depth of additional conv before box predictor: 0
WARNING:tensorflow:From /home/hyebin/.local/lib/python3.6/site-packages/object_detection-0.1-py3.6.egg/object_detection/core/box_list_ops.py:687: add_dispatch_support..wrapper (from tensorflow.python.ops.array_ops) is deprecated and will be removed in a future version.
Instructions for updating:
Use tf.where in 2.0, which has the same broadcast rule as np.where
WARNING:tensorflow:From /home/hyebin/.local/lib/python3.6/site-packages/object_detection-0.1-py3.6.egg/object_detection/exporter.py:356: get_or_create_global_step (from tensorflow.contrib.framework.python.ops.variables) is deprecated and will be removed in a future version.
Instructions for updating:
Please switch to tf.train.get_or_create_global_step
WARNING:tensorflow:From /home/hyebin/.local/lib/python3.6/site-packages/object_detection-0.1-py3.6.egg/object_detection/exporter.py:402: The name tf.train.Saver is deprecated. Please use tf.compat.v1.train.Saver instead.

2020-03-23 20:54:56.959240: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:972] ARM64 does not support NUMA - returning NUMA node zero
2020-03-23 20:54:56.959617: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1640] Found device 0 with properties:
name: NVIDIA Tegra X2 major: 6 minor: 2 memoryClockRate(GHz): 1.02
pciBusID: 0000:00:00.0
2020-03-23 20:54:56.959781: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcudart.so.10.0
2020-03-23 20:54:56.959897: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcublas.so.10.0
2020-03-23 20:54:56.960005: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcufft.so.10.0
2020-03-23 20:54:56.960099: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcurand.so.10.0
2020-03-23 20:54:56.960213: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcusolver.so.10.0
2020-03-23 20:54:56.960307: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcusparse.so.10.0
2020-03-23 20:54:56.960396: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcudnn.so.7
2020-03-23 20:54:56.960741: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:972] ARM64 does not support NUMA - returning NUMA node zero
2020-03-23 20:54:56.961255: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:972] ARM64 does not support NUMA - returning NUMA node zero
2020-03-23 20:54:56.961508: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1763] Adding visible gpu devices: 0
2020-03-23 20:54:56.966720: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:972] ARM64 does not support NUMA - returning NUMA node zero
2020-03-23 20:54:56.967063: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1640] Found device 0 with properties:
name: NVIDIA Tegra X2 major: 6 minor: 2 memoryClockRate(GHz): 1.02
pciBusID: 0000:00:00.0
2020-03-23 20:54:56.967214: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcudart.so.10.0
2020-03-23 20:54:56.967323: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcublas.so.10.0
2020-03-23 20:54:56.967424: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcufft.so.10.0
2020-03-23 20:54:56.967519: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcurand.so.10.0
2020-03-23 20:54:56.967613: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcusolver.so.10.0
2020-03-23 20:54:56.967706: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcusparse.so.10.0
2020-03-23 20:54:56.967798: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcudnn.so.7
2020-03-23 20:54:56.968153: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:972] ARM64 does not support NUMA - returning NUMA node zero
2020-03-23 20:54:56.968653: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:972] ARM64 does not support NUMA - returning NUMA node zero
2020-03-23 20:54:56.968881: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1763] Adding visible gpu devices: 0
2020-03-23 20:54:56.968984: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1181] Device interconnect StreamExecutor with strength 1 edge matrix:
2020-03-23 20:54:56.969040: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1187] 0
2020-03-23 20:54:56.969086: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1200] 0: N
2020-03-23 20:54:56.969442: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:972] ARM64 does not support NUMA - returning NUMA node zero
2020-03-23 20:54:56.969970: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:972] ARM64 does not support NUMA - returning NUMA node zero
2020-03-23 20:54:56.970235: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1326] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 269 MB memory) -> physical GPU (device: 0, name: NVIDIA Tegra X2, pci bus id: 0000:00:00.0, compute capability: 6.2)
WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/tensorflow/python/training/saver.py:1276: checkpoint_exists (from tensorflow.python.training.checkpoint_management) is deprecated and will be removed in a future version.
Instructions for updating:
Use standard file APIs to check for files with this prefix.
INFO:tensorflow:Restoring parameters from data/ssd_mobilenet_v2_egohands/model.ckpt-10000
2020-03-23 20:54:59.980131: W tensorflow/core/framework/op_kernel.cc:1502] OP_REQUIRES failed at save_restore_v2_ops.cc:184 : Not found: Key FeatureExtractor/MobilenetV2/layer_19_2_Conv2d_2_3x3_s2_512_depthwise/BatchNorm/beta not found in checkpoint
Traceback (most recent call last):
File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/client/session.py", line 1356, in _do_call
return fn(*args)
File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/client/session.py", line 1341, in _run_fn
options, feed_dict, fetch_list, target_list, run_metadata)
File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/client/session.py", line 1429, in _call_tf_sessionrun
run_metadata)
tensorflow.python.framework.errors_impl.NotFoundError: 2 root error(s) found.
(0) Not found: Key FeatureExtractor/MobilenetV2/layer_19_2_Conv2d_2_3x3_s2_512_depthwise/BatchNorm/beta not found in checkpoint
[[{{node save/RestoreV2}}]]
(1) Not found: Key FeatureExtractor/MobilenetV2/layer_19_2_Conv2d_2_3x3_s2_512_depthwise/BatchNorm/beta not found in checkpoint
[[{{node save/RestoreV2}}]]
[[save/RestoreV2/_301]]
0 successful operations.
0 derived errors ignored.

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/training/saver.py", line 1286, in restore
{self.saver_def.filename_tensor_name: save_path})
File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/client/session.py", line 950, in run
run_metadata_ptr)
File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/client/session.py", line 1173, in _run
feed_dict_tensor, options, run_metadata)
File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/client/session.py", line 1350, in _do_run
run_metadata)
File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/client/session.py", line 1370, in _do_call
raise type(e)(node_def, op, message)
tensorflow.python.framework.errors_impl.NotFoundError: 2 root error(s) found.
(0) Not found: Key FeatureExtractor/MobilenetV2/layer_19_2_Conv2d_2_3x3_s2_512_depthwise/BatchNorm/beta not found in checkpoint
[[node save/RestoreV2 (defined at /home/hyebin/.local/lib/python3.6/site-packages/object_detection-0.1-py3.6.egg/object_detection/exporter.py:317) ]]
(1) Not found: Key FeatureExtractor/MobilenetV2/layer_19_2_Conv2d_2_3x3_s2_512_depthwise/BatchNorm/beta not found in checkpoint
[[node save/RestoreV2 (defined at /home/hyebin/.local/lib/python3.6/site-packages/object_detection-0.1-py3.6.egg/object_detection/exporter.py:317) ]]
[[save/RestoreV2/_301]]
0 successful operations.
0 derived errors ignored.

Original stack trace for 'save/RestoreV2':
File "camera_tf_trt.py", line 199, in
main()
File "camera_tf_trt.py", line 158, in main
build_trt_pb(args.model, pb_path)
File "/home/hyebin/tf_trt_models/utils/od_utils.py", line 55, in build_trt_pb
checkpoint=checkpoint_path
File "/home/hyebin/tf_trt_models/tf_trt_models/detection.py", line 165, in build_detection_graph
input_shape=[batch_size, None, None, 3]
File "/home/hyebin/.local/lib/python3.6/site-packages/object_detection-0.1-py3.6.egg/object_detection/exporter.py", line 474, in export_inference_graph
write_inference_graph=write_inference_graph)
File "/home/hyebin/.local/lib/python3.6/site-packages/object_detection-0.1-py3.6.egg/object_detection/exporter.py", line 409, in _export_inference_graph
trained_checkpoint_prefix=checkpoint_to_use)
File "/home/hyebin/.local/lib/python3.6/site-packages/object_detection-0.1-py3.6.egg/object_detection/exporter.py", line 317, in write_graph_and_checkpoint
tf.import_graph_def(inference_graph_def, name='')
File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/util/deprecation.py", line 507, in new_func
return func(*args, **kwargs)
File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/framework/importer.py", line 443, in import_graph_def
_ProcessNewOps(graph)
File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/framework/importer.py", line 236, in _ProcessNewOps
for new_op in graph._add_new_tf_operations(compute_devices=False): # pylint: disable=protected-access
File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/framework/ops.py", line 3751, in _add_new_tf_operations
for c_op in c_api_util.new_tf_operations(self)
File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/framework/ops.py", line 3751, in
for c_op in c_api_util.new_tf_operations(self)
File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/framework/ops.py", line 3641, in _create_op_from_tf_operation
ret = Operation(c_op, self)
File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/framework/ops.py", line 2005, in init
self._traceback = tf_stack.extract_stack()

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/training/saver.py", line 1296, in restore
names_to_keys = object_graph_key_mapping(save_path)
File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/training/saver.py", line 1614, in object_graph_key_mapping
object_graph_string = reader.get_tensor(trackable.OBJECT_GRAPH_PROTO_KEY)
File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/pywrap_tensorflow_internal.py", line 678, in get_tensor
return CheckpointReader_GetTensor(self, compat.as_bytes(tensor_str))
tensorflow.python.framework.errors_impl.NotFoundError: Key _CHECKPOINTABLE_OBJECT_GRAPH not found in checkpoint

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "camera_tf_trt.py", line 199, in
main()
File "camera_tf_trt.py", line 158, in main
build_trt_pb(args.model, pb_path)
File "/home/hyebin/tf_trt_models/utils/od_utils.py", line 55, in build_trt_pb
checkpoint=checkpoint_path
File "/home/hyebin/tf_trt_models/tf_trt_models/detection.py", line 165, in build_detection_graph
input_shape=[batch_size, None, None, 3]
File "/home/hyebin/.local/lib/python3.6/site-packages/object_detection-0.1-py3.6.egg/object_detection/exporter.py", line 474, in export_inference_graph
write_inference_graph=write_inference_graph)
File "/home/hyebin/.local/lib/python3.6/site-packages/object_detection-0.1-py3.6.egg/object_detection/exporter.py", line 409, in _export_inference_graph
trained_checkpoint_prefix=checkpoint_to_use)
File "/home/hyebin/.local/lib/python3.6/site-packages/object_detection-0.1-py3.6.egg/object_detection/exporter.py", line 321, in write_graph_and_checkpoint
saver.restore(sess, trained_checkpoint_prefix)
File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/training/saver.py", line 1302, in restore
err, "a Variable name or other graph key that is missing")
tensorflow.python.framework.errors_impl.NotFoundError: Restoring from checkpoint failed. This is most likely due to a Variable name or other graph key that is missing from the checkpoint. Please ensure that you have not altered the graph expected based on the checkpoint. Original error:

2 root error(s) found.
(0) Not found: Key FeatureExtractor/MobilenetV2/layer_19_2_Conv2d_2_3x3_s2_512_depthwise/BatchNorm/beta not found in checkpoint
[[node save/RestoreV2 (defined at /home/hyebin/.local/lib/python3.6/site-packages/object_detection-0.1-py3.6.egg/object_detection/exporter.py:317) ]]
(1) Not found: Key FeatureExtractor/MobilenetV2/layer_19_2_Conv2d_2_3x3_s2_512_depthwise/BatchNorm/beta not found in checkpoint
[[node save/RestoreV2 (defined at /home/hyebin/.local/lib/python3.6/site-packages/object_detection-0.1-py3.6.egg/object_detection/exporter.py:317) ]]
[[save/RestoreV2/_301]]
0 successful operations.
0 derived errors ignored.

Original stack trace for 'save/RestoreV2':
File "camera_tf_trt.py", line 199, in
main()
File "camera_tf_trt.py", line 158, in main
build_trt_pb(args.model, pb_path)
File "/home/hyebin/tf_trt_models/utils/od_utils.py", line 55, in build_trt_pb
checkpoint=checkpoint_path
File "/home/hyebin/tf_trt_models/tf_trt_models/detection.py", line 165, in build_detection_graph
input_shape=[batch_size, None, None, 3]
File "/home/hyebin/.local/lib/python3.6/site-packages/object_detection-0.1-py3.6.egg/object_detection/exporter.py", line 474, in export_inference_graph
write_inference_graph=write_inference_graph)
File "/home/hyebin/.local/lib/python3.6/site-packages/object_detection-0.1-py3.6.egg/object_detection/exporter.py", line 409, in _export_inference_graph
trained_checkpoint_prefix=checkpoint_to_use)
File "/home/hyebin/.local/lib/python3.6/site-packages/object_detection-0.1-py3.6.egg/object_detection/exporter.py", line 317, in write_graph_and_checkpoint
tf.import_graph_def(inference_graph_def, name='')
File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/util/deprecation.py", line 507, in new_func
return func(*args, **kwargs)
File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/framework/importer.py", line 443, in import_graph_def
_ProcessNewOps(graph)
File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/framework/importer.py", line 236, in _ProcessNewOps
for new_op in graph._add_new_tf_operations(compute_devices=False): # pylint: disable=protected-access
File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/framework/ops.py", line 3751, in _add_new_tf_operations
for c_op in c_api_util.new_tf_operations(self)
File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/framework/ops.py", line 3751, in
for c_op in c_api_util.new_tf_operations(self)
File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/framework/ops.py", line 3641, in _create_op_from_tf_operation
ret = Operation(c_op, self)
File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/framework/ops.py", line 2005, in init
self._traceback = tf_stack.extract_stack()

from hand-detection-tutorial.

I think the following error message is complaining about "some missing values in your model checkpoint file".

tensorflow.python.framework.errors_impl.NotFoundError: 2 root error(s) found.
(0) Not found: Key
FeatureExtractor/MobilenetV2/layer_19_2_Conv2d_2_3x3_s2_512_depthwise/BatchNorm/beta not found in checkpoint
[[{{node save/RestoreV2}}]]
......

On your desktop PC:

1. Did you see any errors when you trained the model (and saved the checkpoints)?
2. Are you able to export your trained model to a "frozen_inference_graph.pb" (using the export.sh script)?

from hand-detection-tutorial.

1. Actually, I don't remember it because it's been a while since I've trained. As far as I remember, there were many errors in the process, but ultimately no errors.
   2.How to export "frozen_infernce_graph.pb" here? I do have "frozen_inference_graph.pb"

from hand-detection-tutorial.

Please refer to the Testing the trained model with an image section of my tutorial for how to export a trained model to "frozen_inference_graph.pb".

You are also using tensorflow-1.14.0 (not 2.x) on the desktop PC, right?

from hand-detection-tutorial.

I am able to export "frozen_inference_graph.pb" from trained model.

desktop PC's tensorflow version is 1.8.0.
Jetson TX2's tensorflow version is 1.14.0

from hand-detection-tutorial.

I think the problem is likely due to incompatibility between tensorflow-1.8.0 and 1.14.0. More specifically, the implementation of BatchNorm layer might have been significantly changed between these 2 versions.

You could try these trained model checkpoints instead. As I remember, they were trained with tensorflow-1.11.0.

#21 (comment)

from hand-detection-tutorial.

Thank you. I'll try again by making the tensorflow version the same.

from hand-detection-tutorial.