hikapok / ssd.tensorflow Goto Github PK
View Code? Open in Web Editor NEWState-of-the-art Single Shot MultiBox Detector in Pure TensorFlow, QQ Group: 758790869
License: Apache License 2.0
State-of-the-art Single Shot MultiBox Detector in Pure TensorFlow, QQ Group: 758790869
License: Apache License 2.0
Hi, I have a question of evaluation the ssd300 using different size of images
The layer_steps=[8, 16, 32, 64, 100, 300]
and the layers_shapes = [(38, 38), (19, 19), (10, 10), (5, 5), (3, 3), (1, 1)],
when using the input size 300.
When I evaluate a new image with size 512, the layers_shapes will be
[(64, 64), (32,32), (16, 16), (8, 8), (6, 6), (4, 4)]. Should the "layer_steps" be [8, 16, 32, 64, 85.3, 128]?
Hi,I found that ,your the number of anchors in your code are{3,5,5,5,3,3],but the number of anchors which proposed on paper are[4,6,6,6,4,4],but your code had a better result than paper.It is amazing!
I make my own dateset, and check carefully to ensure all the training image do have at least one bbox. I also set scaffold = None, but it still doesn't work. The ce loss is high, so the total loss is high, and the model diverged with loss = Nan. How can I solve this problem, I have been working with it for some days.
你好!@HiKapok: 向你请教一个问题:
使用你的SSD代码训练完成后,现在需要将其ckpt转换为pb形式,需要output_node_names 但是我发现ssd模型训练的输出好像是
all_labels = tf.concat(labels_list, axis=0)
all_scores = tf.concat(scores_list, axis=0)
all_bboxes = tf.concat(bboxes_list, axis=0)
多个输出,但是,我没办法看到网络Graph的最终输出节点?可以帮忙解答下当前网络的output_node_names是什么吗? 十分感谢
when resize the image to (300,300),you should resize the bbox too, but I can't find it
the convert_tfrecords.py: row 216
anna_file = os.path.join(directory, cur_record[0], 'Annotations', cur_record[1].replace('jpg', 'xml'))
Suggestion as followed:
anna_file = os.path.join(directory, cur_record[0], 'Annotations', cur_record[1].replace('.jpg', '.xml'))
because the file name contain the substring 'jpg' maybe replace not only the last substring.
File "/mnt/lustre/wuguangbin/Data_t1/cornell_ssd_code/Hikapok_SSD/SSD/train_ssd.py", line 463, in main
hooks=[logging_hook], max_steps=FLAGS.max_number_of_steps)
File "/mnt/lustre/share/anaconda2_bigvideo/lib/python2.7/site-packages/tensorflow/python/estimator/estimator.py", line 302, in train
loss = self._train_model(input_fn, hooks, saving_listeners)
File "/mnt/lustre/share/anaconda2_bigvideo/lib/python2.7/site-packages/tensorflow/python/estimator/estimator.py", line 783, in _train_model
_, loss = mon_sess.run([estimator_spec.train_op, estimator_spec.loss])
File "/mnt/lustre/share/anaconda2_bigvideo/lib/python2.7/site-packages/tensorflow/python/training/monitored_session.py", line 537, in exit
self._close_internal(exception_type)
File "/mnt/lustre/share/anaconda2_bigvideo/lib/python2.7/site-packages/tensorflow/python/training/monitored_session.py", line 574, in _close_internal
self._sess.close()
File "/mnt/lustre/share/anaconda2_bigvideo/lib/python2.7/site-packages/tensorflow/python/training/monitored_session.py", line 820, in close
self._sess.close()
File "/mnt/lustre/share/anaconda2_bigvideo/lib/python2.7/site-packages/tensorflow/python/training/monitored_session.py", line 941, in close
ignore_live_threads=True)
File "/mnt/lustre/share/anaconda2_bigvideo/lib/python2.7/site-packages/tensorflow/python/training/coordinator.py", line 389, in join
six.reraise(*self._exc_info_to_raise)
File "/mnt/lustre/share/anaconda2_bigvideo/lib/python2.7/site-packages/tensorflow/python/training/queue_runner_impl.py", line 238, in _run
enqueue_callable()
File "/mnt/lustre/share/anaconda2_bigvideo/lib/python2.7/site-packages/tensorflow/python/client/session.py", line 1231, in _single_operation_run
target_list_as_strings, status, None)
File "/mnt/lustre/share/anaconda2_bigvideo/lib/python2.7/site-packages/tensorflow/python/framework/errors_impl.py", line 473, in exit
c_api.TF_GetCode(self.status.status))
tensorflow.python.framework.errors_impl.DataLossError: corrupted record at 12
[[Node: dataset_data_provider/parallel_read/ReaderReadV2 = ReaderReadV2[device="/job:localhost/replica:0/task:0/device:CPU:0"](dataset_data_provider/parallel_read/TFRecordReaderV2, dataset_data$
rovider/parallel_read/filenames)]]
Hi,writer !I have some problem on your code about how to create anchors.I read your code but I found that the the return of function get_layer_anchors are
tf.expand_dims(y_on_image,** axis=-1), tf.expand_dims(x_on_image, axis=-1),
tf.constant(list_h_on_image, dtype=tf.float32),
tf.constant(list_w_on_image, dtype=tf.float32),
num_anchors_along_depth,
num_anchors_along_spatial
but the shape of list_h_on_image in the first feature layer is ### 4,while the ### shape of the y_on_image is ### 1440(38*38) how they combine with each other? I cannot find the code.
您好 我想使用预训练模型来训练自己的数据 请问您提供的预训练模型是基于channel first训练出来的吗
那我想训练channel last模式的就不能使用该模型吗?
抱歉,我想请问一下,我不用预训练参数可以吗,因为我前面的模型改了,预训练参数对不上,我想自己训练
Hi, I know your SSD implementation from the "Issues" under balancap's SSD implementation and I am training your implementation now when I am typing the questions :D . Good work of your code!
I find out that you used tf.stop_gradient() when you defined your mask in loss. Is it affect the training if not using it?
The implementation by balancap is not stable during training while your implementation is easy to converge. Have you find out the reason?
你好,我想改变一下anchor的个数,比如增加更多的长宽比,请问我应该修改哪些地方呢?谢谢作者!
Hi writer, your job is wonderful ,but how can I get the recall rate in test?Thank you for your answer!
Hi, HiKapok:
Your code is so pretty, good job . I use voc2012test instead of voc2007test, but it's mp is so low, I don't understand the reason.
AP for pottedplant = 0.0000
AP for motorbike = 0.0000
AP for bottle = 0.0000
AP for cow = 0.0000
AP for tvmonitor = 0.0000
AP for person = 0.8002
AP for chair = 0.0008
AP for bus = 0.0000
AP for train = 0.0000
AP for sofa = 0.0000
AP for sheep = 0.0000
AP for boat = 0.0000
AP for horse = 0.0000
AP for diningtable = 0.0078
AP for cat = 0.0000
AP for bicycle = 0.0000
AP for aeroplane = 0.0000
AP for car = 0.0000
AP for dog = 0.0000
AP for bird = 0.0000
Mean AP = 0.0404
i have modified convert_tfrecords.py to produce my own *.tfrecords. My xml file is constructed like this.
when i start traning , i have met problem, and information on console is like this.
ssh://[email protected]:22/home/jiawen/anaconda3/envs/mnist_test/bin/python3.5 -u /home/jiawen/gitcode/ssd.tensorflow/train_ssd.py
2018-07-04 15:41:59.453717: I tensorflow/core/platform/cpu_feature_guard.cc:140] Your CPU supports instructions that this TensorFlow binary was not compiled to use: SSE4.1 SSE4.2 AVX AVX2 FMA
2018-07-04 15:41:59.962936: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:898] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2018-07-04 15:41:59.964121: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1356] Found device 0 with properties:
name: Tesla P100-PCIE-16GB major: 6 minor: 0 memoryClockRate(GHz): 1.3285
pciBusID: 0000:00:08.0
totalMemory: 15.90GiB freeMemory: 15.61GiB
2018-07-04 15:41:59.964159: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1435] Adding visible gpu devices: 0
2018-07-04 15:42:00.289100: I tensorflow/core/common_runtime/gpu/gpu_device.cc:923] Device interconnect StreamExecutor with strength 1 edge matrix:
2018-07-04 15:42:00.289164: I tensorflow/core/common_runtime/gpu/gpu_device.cc:929] 0
2018-07-04 15:42:00.289179: I tensorflow/core/common_runtime/gpu/gpu_device.cc:942] 0: N
2018-07-04 15:42:00.289601: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1053] Created TensorFlow device (/device:GPU:0 with 15135 MB memory) -> physical GPU (device: 0, name: Tesla P100-PCIE-16GB, pci bus id: 0000:00:08.0, compute capability: 6.0)
INFO:tensorflow:Replicating the model_fn
across ['/device:GPU:0']. Variables are going to be placed on ['/device:GPU:0']. Consolidation device is going to be /device:GPU:0.
INFO:tensorflow:Using config: {'_task_type': 'worker', '_num_worker_replicas': 1, '_evaluation_master': '', '_cluster_spec': <tensorflow.python.training.server_lib.ClusterSpec object at 0x7f931a6ca630>, '_keep_checkpoint_every_n_hours': 10000, '_task_id': 0, '_log_step_count_steps': 10, '_global_id_in_cluster': 0, '_save_summary_steps': 500, '_tf_random_seed': 20180503, '_model_dir': './logs/', '_save_checkpoints_secs': 7200, '_service': None, '_save_checkpoints_steps': None, '_train_distribute': None, '_master': '', '_keep_checkpoint_max': 5, '_num_ps_replicas': 0, '_is_chief': True, '_session_config': gpu_options {
per_process_gpu_memory_fraction: 1.0
}
allow_soft_placement: true
}
Starting a training cycle.
INFO:tensorflow:Calling model_fn.
WARNING:tensorflow:From /home/jiawen/gitcode/ssd.tensorflow/net/ssd_net.py:114: calling reduce_sum (from tensorflow.python.ops.math_ops) with keep_dims is deprecated and will be removed in a future version.
Instructions for updating:
keep_dims is deprecated, use keepdims instead
/home/jiawen/anaconda3/envs/mnist_test/lib/python3.5/site-packages/tensorflow/python/ops/gradients_impl.py:100: UserWarning: Converting sparse IndexedSlices to a dense Tensor of unknown shape. This may consume a large amount of memory.
"Converting sparse IndexedSlices to a dense Tensor of unknown shape. "
INFO:tensorflow:Fine-tuning from ./model/vgg16_reducedfc.ckpt. Ignoring missing vars: True.
INFO:tensorflow:Done calling model_fn.
INFO:tensorflow:Create CheckpointSaverHook.
INFO:tensorflow:Graph was finalized.
2018-07-04 15:42:05.769670: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1435] Adding visible gpu devices: 0
2018-07-04 15:42:05.769732: I tensorflow/core/common_runtime/gpu/gpu_device.cc:923] Device interconnect StreamExecutor with strength 1 edge matrix:
2018-07-04 15:42:05.769749: I tensorflow/core/common_runtime/gpu/gpu_device.cc:929] 0
2018-07-04 15:42:05.769763: I tensorflow/core/common_runtime/gpu/gpu_device.cc:942] 0: N
2018-07-04 15:42:05.769994: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1053] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 15135 MB memory) -> physical GPU (device: 0, name: Tesla P100-PCIE-16GB, pci bus id: 0000:00:08.0, compute capability: 6.0)
INFO:tensorflow:Restoring parameters from ./model/vgg16_reducedfc.ckpt
INFO:tensorflow:Running local_init_op.
INFO:tensorflow:Done running local_init_op.
2018-07-04 15:42:13.305340: W tensorflow/core/framework/op_kernel.cc:1318] OP_REQUIRES failed at gather_nd_op.cc:50 : Invalid argument: flat indices[30, :] = [30, -1] does not index into param (shape: [32,6792]).
Traceback (most recent call last):
File "/home/jiawen/anaconda3/envs/mnist_test/lib/python3.5/site-packages/tensorflow/python/client/session.py", line 1322, in _do_call
return fn(*args)
File "/home/jiawen/anaconda3/envs/mnist_test/lib/python3.5/site-packages/tensorflow/python/client/session.py", line 1307, in _run_fn
options, feed_dict, fetch_list, target_list, run_metadata)
File "/home/jiawen/anaconda3/envs/mnist_test/lib/python3.5/site-packages/tensorflow/python/client/session.py", line 1409, in _call_tf_sessionrun
run_metadata)
tensorflow.python.framework.errors_impl.InvalidArgumentError: flat indices[30, :] = [30, -1] does not index into param (shape: [32,6792]).
[[Node: post_forward/GatherNd = GatherNd[Tindices=DT_INT32, Tparams=DT_FLOAT, _device="/job:localhost/replica:0/task:0/device:CPU:0"](post_forward/TopKV2, post_forward/stack)]]
[[Node: post_forward/boolean_mask/Squeeze/_691 = _Recvclient_terminated=false, recv_device="/job:localhost/replica:0/task:0/device:GPU:0", send_device="/job:localhost/replica:0/task:0/device:CPU:0", send_device_incarnation=1, tensor_name="edge_653_post_forward/boolean_mask/Squeeze", tensor_type=DT_INT64, _device="/job:localhost/replica:0/task:0/device:GPU:0"]]
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/home/jiawen/gitcode/ssd.tensorflow/train_ssd.py", line 464, in
tf.app.run()
File "/home/jiawen/anaconda3/envs/mnist_test/lib/python3.5/site-packages/tensorflow/python/platform/app.py", line 126, in run
_sys.exit(main(argv))
File "/home/jiawen/gitcode/ssd.tensorflow/train_ssd.py", line 460, in main
hooks=[logging_hook], max_steps=FLAGS.max_number_of_steps)
File "/home/jiawen/anaconda3/envs/mnist_test/lib/python3.5/site-packages/tensorflow/python/estimator/estimator.py", line 363, in train
loss = self._train_model(input_fn, hooks, saving_listeners)
File "/home/jiawen/anaconda3/envs/mnist_test/lib/python3.5/site-packages/tensorflow/python/estimator/estimator.py", line 843, in _train_model
return self._train_model_default(input_fn, hooks, saving_listeners)
File "/home/jiawen/anaconda3/envs/mnist_test/lib/python3.5/site-packages/tensorflow/python/estimator/estimator.py", line 859, in _train_model_default
saving_listeners)
File "/home/jiawen/anaconda3/envs/mnist_test/lib/python3.5/site-packages/tensorflow/python/estimator/estimator.py", line 1059, in _train_with_estimator_spec
_, loss = mon_sess.run([estimator_spec.train_op, estimator_spec.loss])
File "/home/jiawen/anaconda3/envs/mnist_test/lib/python3.5/site-packages/tensorflow/python/training/monitored_session.py", line 567, in run
run_metadata=run_metadata)
File "/home/jiawen/anaconda3/envs/mnist_test/lib/python3.5/site-packages/tensorflow/python/training/monitored_session.py", line 1043, in run
run_metadata=run_metadata)
File "/home/jiawen/anaconda3/envs/mnist_test/lib/python3.5/site-packages/tensorflow/python/training/monitored_session.py", line 1134, in run
raise six.reraise(*original_exc_info)
File "/home/jiawen/anaconda3/envs/mnist_test/lib/python3.5/site-packages/six.py", line 693, in reraise
raise value
File "/home/jiawen/anaconda3/envs/mnist_test/lib/python3.5/site-packages/tensorflow/python/training/monitored_session.py", line 1119, in run
return self._sess.run(*args, **kwargs)
File "/home/jiawen/anaconda3/envs/mnist_test/lib/python3.5/site-packages/tensorflow/python/training/monitored_session.py", line 1191, in run
run_metadata=run_metadata)
File "/home/jiawen/anaconda3/envs/mnist_test/lib/python3.5/site-packages/tensorflow/python/training/monitored_session.py", line 971, in run
return self._sess.run(*args, **kwargs)
File "/home/jiawen/anaconda3/envs/mnist_test/lib/python3.5/site-packages/tensorflow/python/client/session.py", line 900, in run
run_metadata_ptr)
File "/home/jiawen/anaconda3/envs/mnist_test/lib/python3.5/site-packages/tensorflow/python/client/session.py", line 1135, in _run
feed_dict_tensor, options, run_metadata)
File "/home/jiawen/anaconda3/envs/mnist_test/lib/python3.5/site-packages/tensorflow/python/client/session.py", line 1316, in _do_run
run_metadata)
File "/home/jiawen/anaconda3/envs/mnist_test/lib/python3.5/site-packages/tensorflow/python/client/session.py", line 1335, in _do_call
raise type(e)(node_def, op, message)
tensorflow.python.framework.errors_impl.InvalidArgumentError: flat indices[30, :] = [30, -1] does not index into param (shape: [32,6792]).
[[Node: post_forward/GatherNd = GatherNd[Tindices=DT_INT32, Tparams=DT_FLOAT, _device="/job:localhost/replica:0/task:0/device:CPU:0"](post_forward/TopKV2, post_forward/stack)]]
[[Node: post_forward/boolean_mask/Squeeze/_691 = _Recvclient_terminated=false, recv_device="/job:localhost/replica:0/task:0/device:GPU:0", send_device="/job:localhost/replica:0/task:0/device:CPU:0", send_device_incarnation=1, tensor_name="edge_653_post_forward/boolean_mask/Squeeze", tensor_type=DT_INT64, _device="/job:localhost/replica:0/task:0/device:GPU:0"]]
Caused by op 'post_forward/GatherNd', defined at:
File "/home/jiawen/gitcode/ssd.tensorflow/train_ssd.py", line 464, in
tf.app.run()
File "/home/jiawen/anaconda3/envs/mnist_test/lib/python3.5/site-packages/tensorflow/python/platform/app.py", line 126, in run
_sys.exit(main(argv))
File "/home/jiawen/gitcode/ssd.tensorflow/train_ssd.py", line 460, in main
hooks=[logging_hook], max_steps=FLAGS.max_number_of_steps)
File "/home/jiawen/anaconda3/envs/mnist_test/lib/python3.5/site-packages/tensorflow/python/estimator/estimator.py", line 363, in train
loss = self._train_model(input_fn, hooks, saving_listeners)
File "/home/jiawen/anaconda3/envs/mnist_test/lib/python3.5/site-packages/tensorflow/python/estimator/estimator.py", line 843, in _train_model
return self._train_model_default(input_fn, hooks, saving_listeners)
File "/home/jiawen/anaconda3/envs/mnist_test/lib/python3.5/site-packages/tensorflow/python/estimator/estimator.py", line 856, in _train_model_default
features, labels, model_fn_lib.ModeKeys.TRAIN, self.config)
File "/home/jiawen/anaconda3/envs/mnist_test/lib/python3.5/site-packages/tensorflow/python/estimator/estimator.py", line 831, in _call_model_fn
model_fn_results = self._model_fn(features=features, **kwargs)
File "/home/jiawen/gitcode/ssd.tensorflow/tf_replicate_model_fn.py", line 201, in single_device_model_fn
local_ps_devices=ps_devices)[0] # One device, so one spec is out.
File "/home/jiawen/gitcode/ssd.tensorflow/tf_replicate_model_fn.py", line 534, in _get_loss_towers
**optional_params)
File "/home/jiawen/gitcode/ssd.tensorflow/train_ssd.py", line 321, in ssd_model_fn
score_at_k = tf.gather_nd(topk_prob_for_bg, tf.stack([tf.range(tf.shape(features)[0]), batch_n_neg_select - 1], axis=-1))
File "/home/jiawen/anaconda3/envs/mnist_test/lib/python3.5/site-packages/tensorflow/python/ops/gen_array_ops.py", line 2975, in gather_nd
"GatherNd", params=params, indices=indices, name=name)
File "/home/jiawen/anaconda3/envs/mnist_test/lib/python3.5/site-packages/tensorflow/python/framework/op_def_library.py", line 787, in _apply_op_helper
op_def=op_def)
File "/home/jiawen/anaconda3/envs/mnist_test/lib/python3.5/site-packages/tensorflow/python/framework/ops.py", line 3392, in create_op
op_def=op_def)
File "/home/jiawen/anaconda3/envs/mnist_test/lib/python3.5/site-packages/tensorflow/python/framework/ops.py", line 1718, in init
self._traceback = self._graph._extract_stack() # pylint: disable=protected-access
InvalidArgumentError (see above for traceback): flat indices[30, :] = [30, -1] does not index into param (shape: [32,6792]).
[[Node: post_forward/GatherNd = GatherNd[Tindices=DT_INT32, Tparams=DT_FLOAT, _device="/job:localhost/replica:0/task:0/device:CPU:0"](post_forward/TopKV2, post_forward/stack)]]
[[Node: post_forward/boolean_mask/Squeeze/_691 = _Recvclient_terminated=false, recv_device="/job:localhost/replica:0/task:0/device:GPU:0", send_device="/job:localhost/replica:0/task:0/device:CPU:0", send_device_incarnation=1, tensor_name="edge_653_post_forward/boolean_mask/Squeeze", tensor_type=DT_INT64, _device="/job:localhost/replica:0/task:0/device:GPU:0"]]
Process finished with exit code 1
between_mask = tf.logical_and(tf.less(match_values, high_thres), tf.greater_equal(match_values, low_thres))
negative_mask = less_mask if ignore_between else between_mask
ignore_mask = between_mask if ignore_between else less_mask
what does the between_mask means ? can u explain it more clearly?
Thanks you very much!
Hi, i am curious how did you make the VGG pre-trained weight as it seems different with the default weight provided in tensorflow (tf.slim). Did you apply subsampling for FC6 and FC7? Or did you convert the weights from the caffe model? Thanks in advance.
Hi @HiKapok :
我在使用你的SSD算法应用时候,运行Pascal VOC数据正常!程序可以完美的跑起来,但是当我运行自己的数据集时候,程序卡住同时也不报错!想请教一下,这部分该如何在程序中定位? ps: 数据通过VoC格式已经成功转换为tf格式,没有报错!
数据集内容:其中有部分没有目标bbox坐标信息,只有基本的长宽高!
this repo :
Download the pre-trained VGG-16 model from here and put them into one sub-directory named 'model'.
but I can't get that model, permission denied?
Thanks for your works, it is reproduced easily. There are still some questions confusing me.
The pre-trained vgg16_reduced_fc is different the orginal vgg16, including different weights on conv1/conv1_1 and reducing the fc8 layer, why?
The learning rate is a piecewise function. how did you design this function? and how to set the initial learning rate?
i trained the net in a pc with two 1080ti. the tow gpu is both used. After 120000 steps, the mAP is 0.78. And the log print in the screen like that:
INFO:tensorflow: global_step/sec: 2.22489
INFO: tensorflow: lr=0.00001, ce= 1.487573, loc=0.481032, loss=7.578554, l2=5.609949, acc= 0.835833
INFO: tensorflow: loss=10.1866, step=119990(4.494 sec).
there are two data named "loss", what is their difference?
suppose the second "loss" is the total loss just as the file loss.JPG saved in the log directory. why total loss in your loss.JPG is converged about 7~8. Then my is converged 9~10.
And my time of training one batch cost more than you(yours are 1s) why?
please help me.
你的程序必须提供预训练模型才能训练吗?
i solved the problem mentioned yesterday, the num_classes set to be 3, and change VOC_LABELS to MY_LABELS
but i get another bug
ssh://[email protected]:22/home/jiawen/anaconda3/envs/mnist_test/bin/python3.5 -u /home/jiawen/gitcode/ssd.tensorflow/train_ssd.py
2018-07-05 15:28:46.450990: I tensorflow/core/platform/cpu_feature_guard.cc:140] Your CPU supports instructions that this TensorFlow binary was not compiled to use: SSE4.1 SSE4.2 AVX AVX2 FMA
2018-07-05 15:28:46.940678: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:898] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2018-07-05 15:28:46.941827: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1356] Found device 0 with properties:
name: Tesla P100-PCIE-16GB major: 6 minor: 0 memoryClockRate(GHz): 1.3285
pciBusID: 0000:00:08.0
totalMemory: 15.90GiB freeMemory: 15.61GiB
2018-07-05 15:28:46.941862: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1435] Adding visible gpu devices: 0
2018-07-05 15:28:47.291170: I tensorflow/core/common_runtime/gpu/gpu_device.cc:923] Device interconnect StreamExecutor with strength 1 edge matrix:
2018-07-05 15:28:47.291233: I tensorflow/core/common_runtime/gpu/gpu_device.cc:929] 0
2018-07-05 15:28:47.291253: I tensorflow/core/common_runtime/gpu/gpu_device.cc:942] 0: N
2018-07-05 15:28:47.291677: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1053] Created TensorFlow device (/device:GPU:0 with 15135 MB memory) -> physical GPU (device: 0, name: Tesla P100-PCIE-16GB, pci bus id: 0000:00:08.0, compute capability: 6.0)
INFO:tensorflow:Replicating the model_fn
across ['/device:GPU:0']. Variables are going to be placed on ['/device:GPU:0']. Consolidation device is going to be /device:GPU:0.
INFO:tensorflow:Using config: {'_evaluation_master': '', '_model_dir': './logs/', '_num_worker_replicas': 1, '_session_config': gpu_options {
per_process_gpu_memory_fraction: 1.0
}
allow_soft_placement: true
, '_task_id': 0, '_master': '', '_task_type': 'worker', '_global_id_in_cluster': 0, '_cluster_spec': <tensorflow.python.training.server_lib.ClusterSpec object at 0x7f791f7da630>, '_is_chief': True, '_keep_checkpoint_every_n_hours': 10000, '_keep_checkpoint_max': 5, '_save_checkpoints_secs': 7200, '_log_step_count_steps': 10, '_num_ps_replicas': 0, '_tf_random_seed': 20180503, '_service': None, '_train_distribute': None, '_save_summary_steps': 500, '_save_checkpoints_steps': None}
Starting a training cycle.
INFO:tensorflow:Calling model_fn.
WARNING:tensorflow:From /home/jiawen/gitcode/ssd.tensorflow/net/ssd_net.py:114: calling reduce_sum (from tensorflow.python.ops.math_ops) with keep_dims is deprecated and will be removed in a future version.
Instructions for updating:
keep_dims is deprecated, use keepdims instead
/home/jiawen/anaconda3/envs/mnist_test/lib/python3.5/site-packages/tensorflow/python/ops/gradients_impl.py:100: UserWarning: Converting sparse IndexedSlices to a dense Tensor of unknown shape. This may consume a large amount of memory.
"Converting sparse IndexedSlices to a dense Tensor of unknown shape. "
INFO:tensorflow:Ignoring --checkpoint_path because a checkpoint already exists in ./logs/.
INFO:tensorflow:Done calling model_fn.
INFO:tensorflow:Create CheckpointSaverHook.
INFO:tensorflow:Graph was finalized.
2018-07-05 15:28:52.828754: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1435] Adding visible gpu devices: 0
2018-07-05 15:28:52.828818: I tensorflow/core/common_runtime/gpu/gpu_device.cc:923] Device interconnect StreamExecutor with strength 1 edge matrix:
2018-07-05 15:28:52.828834: I tensorflow/core/common_runtime/gpu/gpu_device.cc:929] 0
2018-07-05 15:28:52.828849: I tensorflow/core/common_runtime/gpu/gpu_device.cc:942] 0: N
2018-07-05 15:28:52.829054: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1053] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 15135 MB memory) -> physical GPU (device: 0, name: Tesla P100-PCIE-16GB, pci bus id: 0000:00:08.0, compute capability: 6.0)
INFO:tensorflow:Restoring parameters from ./logs/model.ckpt-1
Traceback (most recent call last):
File "/home/jiawen/anaconda3/envs/mnist_test/lib/python3.5/site-packages/tensorflow/python/client/session.py", line 1322, in _do_call
return fn(*args)
File "/home/jiawen/anaconda3/envs/mnist_test/lib/python3.5/site-packages/tensorflow/python/client/session.py", line 1307, in _run_fn
options, feed_dict, fetch_list, target_list, run_metadata)
File "/home/jiawen/anaconda3/envs/mnist_test/lib/python3.5/site-packages/tensorflow/python/client/session.py", line 1409, in _call_tf_sessionrun
run_metadata)
tensorflow.python.framework.errors_impl.InvalidArgumentError: Assign requires shapes of both tensors to match. lhs shape= [9] rhs shape= [63]
[[Node: save/Assign_116 = Assign[T=DT_FLOAT, _class=["loc:@ssd300/multibox_head/cls_5/bias"], use_locking=true, validate_shape=true, _device="/job:localhost/replica:0/task:0/device:GPU:0"](ssd300/multibox_head/cls_5/bias/Momentum, save/RestoreV2_1/_1)]]
[[Node: save/RestoreV2_1/_254 = _SendT=DT_FLOAT, client_terminated=false, recv_device="/job:localhost/replica:0/task:0/device:GPU:0", send_device="/job:localhost/replica:0/task:0/device:CPU:0", send_device_incarnation=1, tensor_name="edge_262_save/RestoreV2_1", _device="/job:localhost/replica:0/task:0/device:CPU:0"]]
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/home/jiawen/gitcode/ssd.tensorflow/train_ssd.py", line 464, in
tf.app.run()
File "/home/jiawen/anaconda3/envs/mnist_test/lib/python3.5/site-packages/tensorflow/python/platform/app.py", line 126, in run
_sys.exit(main(argv))
File "/home/jiawen/gitcode/ssd.tensorflow/train_ssd.py", line 460, in main
hooks=[logging_hook], max_steps=FLAGS.max_number_of_steps)
File "/home/jiawen/anaconda3/envs/mnist_test/lib/python3.5/site-packages/tensorflow/python/estimator/estimator.py", line 363, in train
loss = self._train_model(input_fn, hooks, saving_listeners)
File "/home/jiawen/anaconda3/envs/mnist_test/lib/python3.5/site-packages/tensorflow/python/estimator/estimator.py", line 843, in _train_model
return self._train_model_default(input_fn, hooks, saving_listeners)
File "/home/jiawen/anaconda3/envs/mnist_test/lib/python3.5/site-packages/tensorflow/python/estimator/estimator.py", line 859, in _train_model_default
saving_listeners)
File "/home/jiawen/anaconda3/envs/mnist_test/lib/python3.5/site-packages/tensorflow/python/estimator/estimator.py", line 1056, in _train_with_estimator_spec
log_step_count_steps=self._config.log_step_count_steps) as mon_sess:
File "/home/jiawen/anaconda3/envs/mnist_test/lib/python3.5/site-packages/tensorflow/python/training/monitored_session.py", line 405, in MonitoredTrainingSession
stop_grace_period_secs=stop_grace_period_secs)
File "/home/jiawen/anaconda3/envs/mnist_test/lib/python3.5/site-packages/tensorflow/python/training/monitored_session.py", line 816, in init
stop_grace_period_secs=stop_grace_period_secs)
File "/home/jiawen/anaconda3/envs/mnist_test/lib/python3.5/site-packages/tensorflow/python/training/monitored_session.py", line 539, in init
self._sess = _RecoverableSession(self._coordinated_creator)
File "/home/jiawen/anaconda3/envs/mnist_test/lib/python3.5/site-packages/tensorflow/python/training/monitored_session.py", line 1002, in init
_WrappedSession.init(self, self._create_session())
File "/home/jiawen/anaconda3/envs/mnist_test/lib/python3.5/site-packages/tensorflow/python/training/monitored_session.py", line 1007, in _create_session
return self._sess_creator.create_session()
File "/home/jiawen/anaconda3/envs/mnist_test/lib/python3.5/site-packages/tensorflow/python/training/monitored_session.py", line 696, in create_session
self.tf_sess = self._session_creator.create_session()
File "/home/jiawen/anaconda3/envs/mnist_test/lib/python3.5/site-packages/tensorflow/python/training/monitored_session.py", line 467, in create_session
init_fn=self._scaffold.init_fn)
File "/home/jiawen/anaconda3/envs/mnist_test/lib/python3.5/site-packages/tensorflow/python/training/session_manager.py", line 279, in prepare_session
config=config)
File "/home/jiawen/anaconda3/envs/mnist_test/lib/python3.5/site-packages/tensorflow/python/training/session_manager.py", line 207, in _restore_checkpoint
saver.restore(sess, ckpt.model_checkpoint_path)
File "/home/jiawen/anaconda3/envs/mnist_test/lib/python3.5/site-packages/tensorflow/python/training/saver.py", line 1802, in restore
{self.saver_def.filename_tensor_name: save_path})
File "/home/jiawen/anaconda3/envs/mnist_test/lib/python3.5/site-packages/tensorflow/python/client/session.py", line 900, in run
run_metadata_ptr)
File "/home/jiawen/anaconda3/envs/mnist_test/lib/python3.5/site-packages/tensorflow/python/client/session.py", line 1135, in _run
feed_dict_tensor, options, run_metadata)
File "/home/jiawen/anaconda3/envs/mnist_test/lib/python3.5/site-packages/tensorflow/python/client/session.py", line 1316, in _do_run
run_metadata)
File "/home/jiawen/anaconda3/envs/mnist_test/lib/python3.5/site-packages/tensorflow/python/client/session.py", line 1335, in _do_call
raise type(e)(node_def, op, message)
tensorflow.python.framework.errors_impl.InvalidArgumentError: Assign requires shapes of both tensors to match. lhs shape= [9] rhs shape= [63]
[[Node: save/Assign_116 = Assign[T=DT_FLOAT, _class=["loc:@ssd300/multibox_head/cls_5/bias"], use_locking=true, validate_shape=true, _device="/job:localhost/replica:0/task:0/device:GPU:0"](ssd300/multibox_head/cls_5/bias/Momentum, save/RestoreV2_1/_1)]]
[[Node: save/RestoreV2_1/_254 = _SendT=DT_FLOAT, client_terminated=false, recv_device="/job:localhost/replica:0/task:0/device:GPU:0", send_device="/job:localhost/replica:0/task:0/device:CPU:0", send_device_incarnation=1, tensor_name="edge_262_save/RestoreV2_1", _device="/job:localhost/replica:0/task:0/device:CPU:0"]]
Caused by op 'save/Assign_116', defined at:
File "/home/jiawen/gitcode/ssd.tensorflow/train_ssd.py", line 464, in
tf.app.run()
File "/home/jiawen/anaconda3/envs/mnist_test/lib/python3.5/site-packages/tensorflow/python/platform/app.py", line 126, in run
_sys.exit(main(argv))
File "/home/jiawen/gitcode/ssd.tensorflow/train_ssd.py", line 460, in main
hooks=[logging_hook], max_steps=FLAGS.max_number_of_steps)
File "/home/jiawen/anaconda3/envs/mnist_test/lib/python3.5/site-packages/tensorflow/python/estimator/estimator.py", line 363, in train
loss = self._train_model(input_fn, hooks, saving_listeners)
File "/home/jiawen/anaconda3/envs/mnist_test/lib/python3.5/site-packages/tensorflow/python/estimator/estimator.py", line 843, in _train_model
return self._train_model_default(input_fn, hooks, saving_listeners)
File "/home/jiawen/anaconda3/envs/mnist_test/lib/python3.5/site-packages/tensorflow/python/estimator/estimator.py", line 859, in _train_model_default
saving_listeners)
File "/home/jiawen/anaconda3/envs/mnist_test/lib/python3.5/site-packages/tensorflow/python/estimator/estimator.py", line 1056, in _train_with_estimator_spec
log_step_count_steps=self._config.log_step_count_steps) as mon_sess:
File "/home/jiawen/anaconda3/envs/mnist_test/lib/python3.5/site-packages/tensorflow/python/training/monitored_session.py", line 405, in MonitoredTrainingSession
stop_grace_period_secs=stop_grace_period_secs)
File "/home/jiawen/anaconda3/envs/mnist_test/lib/python3.5/site-packages/tensorflow/python/training/monitored_session.py", line 816, in init
stop_grace_period_secs=stop_grace_period_secs)
File "/home/jiawen/anaconda3/envs/mnist_test/lib/python3.5/site-packages/tensorflow/python/training/monitored_session.py", line 539, in init
self._sess = _RecoverableSession(self._coordinated_creator)
File "/home/jiawen/anaconda3/envs/mnist_test/lib/python3.5/site-packages/tensorflow/python/training/monitored_session.py", line 1002, in init
_WrappedSession.init(self, self._create_session())
File "/home/jiawen/anaconda3/envs/mnist_test/lib/python3.5/site-packages/tensorflow/python/training/monitored_session.py", line 1007, in _create_session
return self._sess_creator.create_session()
File "/home/jiawen/anaconda3/envs/mnist_test/lib/python3.5/site-packages/tensorflow/python/training/monitored_session.py", line 696, in create_session
self.tf_sess = self._session_creator.create_session()
File "/home/jiawen/anaconda3/envs/mnist_test/lib/python3.5/site-packages/tensorflow/python/training/monitored_session.py", line 458, in create_session
self._scaffold.finalize()
File "/home/jiawen/anaconda3/envs/mnist_test/lib/python3.5/site-packages/tensorflow/python/training/monitored_session.py", line 214, in finalize
self._saver.build()
File "/home/jiawen/anaconda3/envs/mnist_test/lib/python3.5/site-packages/tensorflow/python/training/saver.py", line 1347, in build
self._build(self._filename, build_save=True, build_restore=True)
File "/home/jiawen/anaconda3/envs/mnist_test/lib/python3.5/site-packages/tensorflow/python/training/saver.py", line 1384, in _build
build_save=build_save, build_restore=build_restore)
File "/home/jiawen/anaconda3/envs/mnist_test/lib/python3.5/site-packages/tensorflow/python/training/saver.py", line 829, in _build_internal
restore_sequentially, reshape)
File "/home/jiawen/anaconda3/envs/mnist_test/lib/python3.5/site-packages/tensorflow/python/training/saver.py", line 525, in _AddShardedRestoreOps
name="restore_shard"))
File "/home/jiawen/anaconda3/envs/mnist_test/lib/python3.5/site-packages/tensorflow/python/training/saver.py", line 494, in _AddRestoreOps
assign_ops.append(saveable.restore(saveable_tensors, shapes))
File "/home/jiawen/anaconda3/envs/mnist_test/lib/python3.5/site-packages/tensorflow/python/training/saver.py", line 185, in restore
self.op.get_shape().is_fully_defined())
File "/home/jiawen/anaconda3/envs/mnist_test/lib/python3.5/site-packages/tensorflow/python/ops/state_ops.py", line 283, in assign
validate_shape=validate_shape)
File "/home/jiawen/anaconda3/envs/mnist_test/lib/python3.5/site-packages/tensorflow/python/ops/gen_state_ops.py", line 60, in assign
use_locking=use_locking, name=name)
File "/home/jiawen/anaconda3/envs/mnist_test/lib/python3.5/site-packages/tensorflow/python/framework/op_def_library.py", line 787, in _apply_op_helper
op_def=op_def)
File "/home/jiawen/anaconda3/envs/mnist_test/lib/python3.5/site-packages/tensorflow/python/framework/ops.py", line 3392, in create_op
op_def=op_def)
File "/home/jiawen/anaconda3/envs/mnist_test/lib/python3.5/site-packages/tensorflow/python/framework/ops.py", line 1718, in init
self._traceback = self._graph._extract_stack() # pylint: disable=protected-access
InvalidArgumentError (see above for traceback): Assign requires shapes of both tensors to match. lhs shape= [9] rhs shape= [63]
[[Node: save/Assign_116 = Assign[T=DT_FLOAT, _class=["loc:@ssd300/multibox_head/cls_5/bias"], use_locking=true, validate_shape=true, _device="/job:localhost/replica:0/task:0/device:GPU:0"](ssd300/multibox_head/cls_5/bias/Momentum, save/RestoreV2_1/_1)]]
[[Node: save/RestoreV2_1/_254 = _SendT=DT_FLOAT, client_terminated=false, recv_device="/job:localhost/replica:0/task:0/device:GPU:0", send_device="/job:localhost/replica:0/task:0/device:CPU:0", send_device_incarnation=1, tensor_name="edge_262_save/RestoreV2_1", _device="/job:localhost/replica:0/task:0/device:CPU:0"]]
Process finished with exit code 1
can you list out all lines which should be changed if i want to train by my own data
Hi! I have a dataset of only one class object. I attempted to use balancap's SSD-Tensorflow but meet a unsolvable trouble, so I have to look for a substitute. Is it easy to change your code for my dataset? What should I notice? Thanks in advance!
I used the pre-trained vgg16 model as you provided ,but an error occured :
INFO:tensorflow:Fine-tuning from None. Ignoring missing vars: True.
Traceback (most recent call last):
File "train_ssd.py", line 464, in
tf.app.run()
File "/home/h/anaconda3/lib/python3.6/site-packages/tensorflow/python/platform/app.py", line 126, in run
_sys.exit(main(argv))
File "train_ssd.py", line 460, in main
hooks=[logging_hook], max_steps=FLAGS.max_number_of_steps)
File "/home/h/anaconda3/lib/python3.6/site-packages/tensorflow/python/estimator/estimator.py", line 352, in train
loss = self._train_model(input_fn, hooks, saving_listeners)
File "/home/h/anaconda3/lib/python3.6/site-packages/tensorflow/python/estimator/estimator.py", line 812, in _train_model
features, labels, model_fn_lib.ModeKeys.TRAIN, self.config)
File "/home/h/anaconda3/lib/python3.6/site-packages/tensorflow/python/estimator/estimator.py", line 793, in _call_model_fn
model_fn_results = self._model_fn(features=features, **kwargs)
File "/home/h/anaconda3/lib/python3.6/site-packages/tensorflow/contrib/estimator/python/estimator/replicate_model_fn.py", line 220, in single_device_model_fn
local_ps_devices=ps_devices)[0] # One device, so one spec is out.
File "/home/h/anaconda3/lib/python3.6/site-packages/tensorflow/contrib/estimator/python/estimator/replicate_model_fn.py", line 558, in _get_loss_towers
**optional_params)
File "train_ssd.py", line 403, in ssd_model_fn
scaffold=tf.train.Scaffold(init_fn=get_init_fn()))
File "train_ssd.py", line 158, in get_init_fn
name_remap={'/kernel': '/weights', '/bias': '/biases'})
File "/home/h/Desktop/SSD.TensorFlow-master/utility/scaffolds.py", line 66, in get_init_fn_for_scaffold
reader = tf.train.NewCheckpointReader(checkpoint_path)
File "/home/h/anaconda3/lib/python3.6/site-packages/tensorflow/python/pywrap_tensorflow_internal.py", line 254, in NewCheckpointReader
return CheckpointReader(compat.as_bytes(filepattern), status)
File "/home/h/anaconda3/lib/python3.6/site-packages/tensorflow/python/util/compat.py", line 67, in as_bytes
(bytes_or_text,))
TypeError: Expected binary or unicode string, got None
could you help me
I trained the VOC dataset, but only got 64% mAP. Is there any trick do you use to reach the report map (77%) without the pertained model?
运行完eeval_ssd,之后运行voc_eval,提示FileNotFoundError: [Errno 2] No such file or directory: '/media/streamax/disk1/DataSet/test-VOC-2012/VOCdevkit/VOC2012/Annotations/2008_000001.xml'
没有2008_0000001.xml啊
1.can this project train my own data?
2.if can,how to train?
thank you for your codes, could you give the training time and your machine hardware configuration? I am also training a ssd300 net on my own dataset, but I found the training process is very slow, so I want to know your training time for 60k steps on pascal voc datasets, that is useful for me, maybe. Thank you!
Hi,
I have changed the model definition for face detection task. As mentioned in readme, I have changed the required places for adapting it in 1.4tf.
But NaN loss issue is not getting solved. Even in reduced learning rate of 1e-4 its giving Nan.
have you found any solution to this?
Hi Changan,
Do you have any experience with tf.contrib.distribute.MirroredStrategy? As TF is going to depreciate replicate_model_fn, I was trying to update your code with MirroredStrategy, but I always got "ValueError: destinations must be one of a DistributedValues
object, a device string, a list of device strings or None".
Please take a look when you have time.
Thanks,
Yu
When trying to run eval_ssd.py I get the following error:
File "eval_ssd.py", line 373, in ssd_model_fn
loss=None, train_op=None)
TypeError: new() got an unexpected keyword argument 'prediction_hooks'
I'm using TF 1.4.1. Is this another TF compatibility issue?
Is there an easy patch I can use as you did for tf_replicate_model_fn?
Thanks in advance!
I find the reason is that your anchor_ratios don't have 1. If the 1 is added then the anchors is 8732 which is match with the original ssd source.
作者你好,我没有很能看懂您代码中数据预处理的随机裁剪部分,请问对一张图片随机裁剪的时候,min_iou 是每次从 ratio_list 中等概率随机得到吗?这是不是意味着极端情况如果我将ratio_list变成[0.1, 0.1, 0.1, 0.1, 0.1, 1.]的话,随机裁剪出的iou>0.1的框是最多的?
还有,ratio_list[-0.1,0.1, 0.3, 0.5, 0.7, 0.9, 1.]有-0.1是为了随机裁剪出只有背景的图片吗?
期待您的解答~
AttributeError: module 'tensorflow.python.layers.layers' has no attribute 'Layer'
Will it be that the TF version is too low?
预训练模型。
I have download vgg16, but there still have a problem. I'm using tensorflow 1.8.
raceback (most recent call last):
File "/root/Downloads/SSD.TensorFlow-master/train_ssd.py", line 464, in
tf.app.run()
File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/platform/app.py", line 126, in run
_sys.exit(main(argv))
File "/root/Downloads/SSD.TensorFlow-master/train_ssd.py", line 460, in main
hooks=[logging_hook], max_steps=FLAGS.max_number_of_steps)
File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/estimator/estimator.py", line 363, in train
loss = self._train_model(input_fn, hooks, saving_listeners)
File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/estimator/estimator.py", line 843, in _train_model
return self._train_model_default(input_fn, hooks, saving_listeners)
File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/estimator/estimator.py", line 856, in _train_model_default
features, labels, model_fn_lib.ModeKeys.TRAIN, self.config)
File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/estimator/estimator.py", line 831, in _call_model_fn
model_fn_results = self._model_fn(features=features, **kwargs)
File "/usr/local/lib/python3.5/dist-packages/tensorflow/contrib/estimator/python/estimator/replicate_model_fn.py", line 221, in single_device_model_fn
local_ps_devices=ps_devices)[0] # One device, so one spec is out.
File "/usr/local/lib/python3.5/dist-packages/tensorflow/contrib/estimator/python/estimator/replicate_model_fn.py", line 559, in _get_loss_towers
**optional_params)
File "/root/Downloads/SSD.TensorFlow-master/train_ssd.py", line 403, in ssd_model_fn
scaffold=tf.train.Scaffold(init_fn=get_init_fn()))
File "/root/Downloads/SSD.TensorFlow-master/train_ssd.py", line 158, in get_init_fn
name_remap={'/kernel': '/weights', '/bias': '/biases'})
File "/root/Downloads/SSD.TensorFlow-master/utility/scaffolds.py", line 66, in get_init_fn_for_scaffold
reader = tf.train.NewCheckpointReader(checkpoint_path)
File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/pywrap_tensorflow_internal.py", line 290, in NewCheckpointReader
return CheckpointReader(compat.as_bytes(filepattern), status)
File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/util/compat.py", line 68, in as_bytes
(bytes_or_text,))
TypeError: Expected binary or unicode string, got None
Hi friend,
I interesting in moving to test your code on COCO dataset. I downloaded the train/val files and ready to go. So, do you have any idea where should I start?
after i download your file and checkpoint ,then i run the eval.py, it give out error like this.
C:\Users\DELL\PycharmProjects\unstructured_data\venv\Scripts\python.exe D:/git_code/SSD.TensorFlow/eval_ssd.py
INFO:tensorflow:Using config: {'_model_dir': './logs/', '_tf_random_seed': None, '_save_summary_steps': 500, '_save_checkpoints_steps': None, '_save_checkpoints_secs': None, '_session_config': gpu_options {
per_process_gpu_memory_fraction: 1.0
}
allow_soft_placement: true
, '_keep_checkpoint_max': 5, '_keep_checkpoint_every_n_hours': 10000, '_log_step_count_steps': 10, '_train_distribute': None, '_service': None, '_cluster_spec': <tensorflow.python.training.server_lib.ClusterSpec object at 0x000002B7C3A76F98>, '_task_type': 'worker', '_task_id': 0, '_global_id_in_cluster': 0, '_master': '', '_evaluation_master': '', '_is_chief': True, '_num_ps_replicas': 0, '_num_worker_replicas': 1}
Starting a predict cycle.
INFO:tensorflow:Calling model_fn.
WARNING:tensorflow:From D:\git_code\SSD.TensorFlow\net\ssd_net.py:114: calling reduce_sum (from tensorflow.python.ops.math_ops) with keep_dims is deprecated and will be removed in a future version.
Instructions for updating:
keep_dims is deprecated, use keepdims instead
INFO:tensorflow:Done calling model_fn.
INFO:tensorflow:Graph was finalized.
2018-06-28 21:01:10.495852: I T:\src\github\tensorflow\tensorflow\core\platform\cpu_feature_guard.cc:140] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2
INFO:tensorflow:Restoring parameters from ./model\vgg16_reducedfc.ckpt
2018-06-28 21:01:10.574406: W T:\src\github\tensorflow\tensorflow\core\framework\op_kernel.cc:1318] OP_REQUIRES failed at save_restore_v2_ops.cc:184 : Not found: Key global_step not found in checkpoint
Traceback (most recent call last):
File "C:\Users\DELL\PycharmProjects\unstructured_data\venv\lib\site-packages\tensorflow\python\client\session.py", line 1322, in _do_call
return fn(*args)
File "C:\Users\DELL\PycharmProjects\unstructured_data\venv\lib\site-packages\tensorflow\python\client\session.py", line 1307, in _run_fn
options, feed_dict, fetch_list, target_list, run_metadata)
File "C:\Users\DELL\PycharmProjects\unstructured_data\venv\lib\site-packages\tensorflow\python\client\session.py", line 1409, in _call_tf_sessionrun
run_metadata)
tensorflow.python.framework.errors_impl.NotFoundError: Key global_step not found in checkpoint
[[Node: save/RestoreV2 = RestoreV2[dtypes=[DT_INT64, DT_FLOAT, DT_FLOAT, DT_FLOAT, DT_FLOAT, ..., DT_FLOAT, DT_FLOAT, DT_FLOAT, DT_FLOAT, DT_FLOAT], _device="/job:localhost/replica:0/task:0/device:CPU:0"](_arg_save/Const_0_0, save/RestoreV2/tensor_names, save/RestoreV2/shape_and_slices)]]
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "D:/git_code/SSD.TensorFlow/eval_ssd.py", line 456, in
tf.app.run()
File "C:\Users\DELL\PycharmProjects\unstructured_data\venv\lib\site-packages\tensorflow\python\platform\app.py", line 126, in run
_sys.exit(main(argv))
File "D:/git_code/SSD.TensorFlow/eval_ssd.py", line 427, in main
det_results = list(pred_results)
File "C:\Users\DELL\PycharmProjects\unstructured_data\venv\lib\site-packages\tensorflow\python\estimator\estimator.py", line 507, in predict
hooks=all_hooks) as mon_sess:
File "C:\Users\DELL\PycharmProjects\unstructured_data\venv\lib\site-packages\tensorflow\python\training\monitored_session.py", line 816, in init
stop_grace_period_secs=stop_grace_period_secs)
File "C:\Users\DELL\PycharmProjects\unstructured_data\venv\lib\site-packages\tensorflow\python\training\monitored_session.py", line 539, in init
self._sess = _RecoverableSession(self._coordinated_creator)
File "C:\Users\DELL\PycharmProjects\unstructured_data\venv\lib\site-packages\tensorflow\python\training\monitored_session.py", line 1002, in init
_WrappedSession.init(self, self._create_session())
File "C:\Users\DELL\PycharmProjects\unstructured_data\venv\lib\site-packages\tensorflow\python\training\monitored_session.py", line 1007, in _create_session
return self._sess_creator.create_session()
File "C:\Users\DELL\PycharmProjects\unstructured_data\venv\lib\site-packages\tensorflow\python\training\monitored_session.py", line 696, in create_session
self.tf_sess = self._session_creator.create_session()
File "C:\Users\DELL\PycharmProjects\unstructured_data\venv\lib\site-packages\tensorflow\python\training\monitored_session.py", line 467, in create_session
init_fn=self._scaffold.init_fn)
File "C:\Users\DELL\PycharmProjects\unstructured_data\venv\lib\site-packages\tensorflow\python\training\session_manager.py", line 279, in prepare_session
config=config)
File "C:\Users\DELL\PycharmProjects\unstructured_data\venv\lib\site-packages\tensorflow\python\training\session_manager.py", line 191, in _restore_checkpoint
saver.restore(sess, checkpoint_filename_with_path)
File "C:\Users\DELL\PycharmProjects\unstructured_data\venv\lib\site-packages\tensorflow\python\training\saver.py", line 1802, in restore
{self.saver_def.filename_tensor_name: save_path})
File "C:\Users\DELL\PycharmProjects\unstructured_data\venv\lib\site-packages\tensorflow\python\client\session.py", line 900, in run
run_metadata_ptr)
File "C:\Users\DELL\PycharmProjects\unstructured_data\venv\lib\site-packages\tensorflow\python\client\session.py", line 1135, in _run
feed_dict_tensor, options, run_metadata)
File "C:\Users\DELL\PycharmProjects\unstructured_data\venv\lib\site-packages\tensorflow\python\client\session.py", line 1316, in _do_run
run_metadata)
File "C:\Users\DELL\PycharmProjects\unstructured_data\venv\lib\site-packages\tensorflow\python\client\session.py", line 1335, in _do_call
raise type(e)(node_def, op, message)
tensorflow.python.framework.errors_impl.NotFoundError: Key global_step not found in checkpoint
[[Node: save/RestoreV2 = RestoreV2[dtypes=[DT_INT64, DT_FLOAT, DT_FLOAT, DT_FLOAT, DT_FLOAT, ..., DT_FLOAT, DT_FLOAT, DT_FLOAT, DT_FLOAT, DT_FLOAT], _device="/job:localhost/replica:0/task:0/device:CPU:0"](_arg_save/Const_0_0, save/RestoreV2/tensor_names, save/RestoreV2/shape_and_slices)]]
Caused by op 'save/RestoreV2', defined at:
File "D:/git_code/SSD.TensorFlow/eval_ssd.py", line 456, in
tf.app.run()
File "C:\Users\DELL\PycharmProjects\unstructured_data\venv\lib\site-packages\tensorflow\python\platform\app.py", line 126, in run
_sys.exit(main(argv))
File "D:/git_code/SSD.TensorFlow/eval_ssd.py", line 427, in main
det_results = list(pred_results)
File "C:\Users\DELL\PycharmProjects\unstructured_data\venv\lib\site-packages\tensorflow\python\estimator\estimator.py", line 507, in predict
hooks=all_hooks) as mon_sess:
File "C:\Users\DELL\PycharmProjects\unstructured_data\venv\lib\site-packages\tensorflow\python\training\monitored_session.py", line 816, in init
stop_grace_period_secs=stop_grace_period_secs)
File "C:\Users\DELL\PycharmProjects\unstructured_data\venv\lib\site-packages\tensorflow\python\training\monitored_session.py", line 539, in init
self._sess = _RecoverableSession(self._coordinated_creator)
File "C:\Users\DELL\PycharmProjects\unstructured_data\venv\lib\site-packages\tensorflow\python\training\monitored_session.py", line 1002, in init
_WrappedSession.init(self, self._create_session())
File "C:\Users\DELL\PycharmProjects\unstructured_data\venv\lib\site-packages\tensorflow\python\training\monitored_session.py", line 1007, in _create_session
return self._sess_creator.create_session()
File "C:\Users\DELL\PycharmProjects\unstructured_data\venv\lib\site-packages\tensorflow\python\training\monitored_session.py", line 696, in create_session
self.tf_sess = self._session_creator.create_session()
File "C:\Users\DELL\PycharmProjects\unstructured_data\venv\lib\site-packages\tensorflow\python\training\monitored_session.py", line 458, in create_session
self._scaffold.finalize()
File "C:\Users\DELL\PycharmProjects\unstructured_data\venv\lib\site-packages\tensorflow\python\training\monitored_session.py", line 212, in finalize
self._saver = training_saver._get_saver_or_default() # pylint: disable=protected-access
File "C:\Users\DELL\PycharmProjects\unstructured_data\venv\lib\site-packages\tensorflow\python\training\saver.py", line 910, in _get_saver_or_default
saver = Saver(sharded=True, allow_empty=True)
File "C:\Users\DELL\PycharmProjects\unstructured_data\venv\lib\site-packages\tensorflow\python\training\saver.py", line 1338, in init
self.build()
File "C:\Users\DELL\PycharmProjects\unstructured_data\venv\lib\site-packages\tensorflow\python\training\saver.py", line 1347, in build
self._build(self._filename, build_save=True, build_restore=True)
File "C:\Users\DELL\PycharmProjects\unstructured_data\venv\lib\site-packages\tensorflow\python\training\saver.py", line 1384, in _build
build_save=build_save, build_restore=build_restore)
File "C:\Users\DELL\PycharmProjects\unstructured_data\venv\lib\site-packages\tensorflow\python\training\saver.py", line 829, in _build_internal
restore_sequentially, reshape)
File "C:\Users\DELL\PycharmProjects\unstructured_data\venv\lib\site-packages\tensorflow\python\training\saver.py", line 525, in _AddShardedRestoreOps
name="restore_shard"))
File "C:\Users\DELL\PycharmProjects\unstructured_data\venv\lib\site-packages\tensorflow\python\training\saver.py", line 472, in _AddRestoreOps
restore_sequentially)
File "C:\Users\DELL\PycharmProjects\unstructured_data\venv\lib\site-packages\tensorflow\python\training\saver.py", line 886, in bulk_restore
return io_ops.restore_v2(filename_tensor, names, slices, dtypes)
File "C:\Users\DELL\PycharmProjects\unstructured_data\venv\lib\site-packages\tensorflow\python\ops\gen_io_ops.py", line 1546, in restore_v2
shape_and_slices=shape_and_slices, dtypes=dtypes, name=name)
File "C:\Users\DELL\PycharmProjects\unstructured_data\venv\lib\site-packages\tensorflow\python\framework\op_def_library.py", line 787, in _apply_op_helper
op_def=op_def)
File "C:\Users\DELL\PycharmProjects\unstructured_data\venv\lib\site-packages\tensorflow\python\framework\ops.py", line 3392, in create_op
op_def=op_def)
File "C:\Users\DELL\PycharmProjects\unstructured_data\venv\lib\site-packages\tensorflow\python\framework\ops.py", line 1718, in init
self._traceback = self._graph._extract_stack() # pylint: disable=protected-access
NotFoundError (see above for traceback): Key global_step not found in checkpoint
[[Node: save/RestoreV2 = RestoreV2[dtypes=[DT_INT64, DT_FLOAT, DT_FLOAT, DT_FLOAT, DT_FLOAT, ..., DT_FLOAT, DT_FLOAT, DT_FLOAT, DT_FLOAT, DT_FLOAT], _device="/job:localhost/replica:0/task:0/device:CPU:0"](_arg_save/Const_0_0, save/RestoreV2/tensor_names, save/RestoreV2/shape_and_slices)]]
Process finished with exit code 1
作者你好,我想在训练中打印出final_mask 的值,但是发现final_mask 是一个tensor,我没办法把它转换成数组打印出来,请问您能在这个问题上帮助我一下吗?我最近也在做SSD的研究,可以加个微信交流一下吗?
image_ids' values are in the format '0.xxx', while class_recs' keys are in the format 'xxxxxxx', so line 22:
R = class_recs[image_ids[d]]
gives a KeyError.
1.What does the acc(post_forward/cls_accuracy_1) mean? Is it possible to show the AP/mAP during training?
2. I try half of the batchsize with GTX1060 and try to lower the learning rate and run more steps as u told. Is it also necessary to change the end_learning_rate/decay_boundaries/lr_deacay_factors?
In ordinary model,I can use the function of sess.run to get the tensors' running value,but how can use tf.estimator to achieve the same goal.Looking forward to your reply,thanks!
hi, your work is nice, but i meet some problems about train my own data which has 77 kinds of fruits&veg. i load the pre-trained vgg16 model and train about 86000 steps, followed pics show the result, could you give some suggestions?
And i wonder how you solve the loss not converge problem(balancap version), looking forward to your reply, thank you
I followed everything as you write,but an error occur :
TypeError: Expected binary or unicode string,got None
can you help me
I said this because the original ssd trained on 16551 images of VOC0712.
your code is very elegant and readable! Following your instruction, I train the
ssd300 model on VOC07+12 dataset,but the mAP on VOC07 test is only 76.0%.
I wonder that you achieve the state of the art performance(77.8%),even better than original paper's norm(77.2%). Did you do extra improvement?
What's the purpose of eval_ssd.py?
I donnot understand what the 'anchor_allowed_borders' is for in function 'encode_all_labels'
inside_mask = tf.logical_and(tf.logical_and(anchors_ymin > -anchor_allowed_borders * 1.,
anchors_xmin > -anchor_allowed_borders * 1.),
tf.logical_and(anchors_ymax < (1. + anchor_allowed_borders * 1.),
anchors_xmax < (1. + anchor_allowed_borders * 1.)))
I read your code that 'anchor_allowed_borders ' is all 1.0, so will the mask always be true? (as the anchors_ymin, anchors_ymax, anchors_xmin, anchors_ymax are always between 0.0 and 1.0
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.