Using config:
{'ANCHOR_SCALES': [16],
'DATA_DIR': '/home/text-detection-ctpn/data',
'DEDUP_BOXES': 0.0625,
'EPS': 1e-14,
'EXP_DIR': 'ctpn_end2end',
'GPU_ID': 0,
'IS_EXTRAPOLATING': True,
'IS_MULTISCALE': False,
'IS_RPN': True,
'LOG_DIR': 'ctpn',
'MATLAB': 'matlab',
'MODELS_DIR': '/home/peng_yuxiang/tensorflow_tests/code/CPTN_Tensorflow/text-detection-ctpn/models/pascal_voc',
'NCLASSES': 2,
'NET_NAME': 'VGGnet',
'PIXEL_MEANS': array([[[ 102.9801, 115.9465, 122.7717]]]),
'REGION_PROPOSAL': 'RPN',
'RNG_SEED': 3,
'ROOT_DIR': '/home/peng_yuxiang/tensorflow_tests/code/CPTN_Tensorflow/text-detection-ctpn',
'SUBCLS_NAME': 'voxel_exemplars',
'TEST': {'BBOX_REG': True,
'HAS_RPN': True,
'MAX_SIZE': 1000,
'NMS': 0.3,
'PROPOSAL_METHOD': 'selective_search',
'RPN_MIN_SIZE': 8,
'RPN_NMS_THRESH': 0.7,
'RPN_POST_NMS_TOP_N': 1000,
'RPN_PRE_NMS_TOP_N': 12000,
'SCALES': [600],
'SVM': False},
'TRAIN': {'ASPECTS': [1],
'ASPECT_GROUPING': True,
'BATCH_SIZE': 300,
'BBOX_INSIDE_WEIGHTS': [1, 1, 1, 1],
'BBOX_NORMALIZE_MEANS': [0.0, 0.0, 0.0, 0.0],
'BBOX_NORMALIZE_STDS': [0.1, 0.1, 0.2, 0.2],
'BBOX_NORMALIZE_TARGETS': True,
'BBOX_NORMALIZE_TARGETS_PRECOMPUTED': True,
'BBOX_REG': True,
'BBOX_THRESH': 0.5,
'BG_THRESH_HI': 0.5,
'BG_THRESH_LO': 0.0,
'DISPLAY': 10,
'DONTCARE_AREA_INTERSECTION_HI': 0.5,
'FG_FRACTION': 0.3,
'FG_THRESH': 0.5,
'GAMMA': 0.1,
'HAS_RPN': True,
'IMS_PER_BATCH': 1,
'KERNEL_SIZE': 5,
'LEARNING_RATE': 0.001,
'LOG_IMAGE_ITERS': 100,
'MAX_SIZE': 1000,
'MOMENTUM': 0.9,
'OHEM': False,
'PRECLUDE_HARD_SAMPLES': True,
'PROPOSAL_METHOD': 'gt',
'RANDOM_DOWNSAMPLE': False,
'RPN_BATCHSIZE': 256,
'RPN_BBOX_INSIDE_WEIGHTS': [1, 1, 1, 1],
'RPN_CLOBBER_POSITIVES': False,
'RPN_FG_FRACTION': 0.5,
'RPN_MIN_SIZE': 8,
'RPN_NEGATIVE_OVERLAP': 0.3,
'RPN_NMS_THRESH': 0.7,
'RPN_POSITIVE_OVERLAP': 0.7,
'RPN_POSITIVE_WEIGHT': -1.0,
'RPN_POST_NMS_TOP_N': 2000,
'RPN_PRE_NMS_TOP_N': 12000,
'SCALES': [600],
'SCALES_BASE': [0.25, 0.5, 1.0, 2.0, 3.0],
'SNAPSHOT_INFIX': '',
'SNAPSHOT_ITERS': 1000,
'SNAPSHOT_PREFIX': 'VGGnet_fast_rcnn',
'SOLVER': 'Momentum',
'STEPSIZE': 50,
'USE_FLIPPED': True,
'USE_PREFETCH': False,
'WEIGHT_DECAY': 0.0005},
'USE_GPU_NMS': True}
<bound method pascal_voc.default_roidb of <lib.datasets.pascal_voc.pascal_voc object at 0x7f04ee1b6910>>
Loaded dataset voc_2007_trainval
for training
Appending horizontally-flipped training examples...
voc_2007_trainval gt roidb loaded from /home/peng_yuxiang/tensorflow_tests/code/CPTN_Tensorflow/text-detection-ctpn/data/cache/voc_2007_trainval_gt_roidb.pkl
done
Preparing training data...
done
Output will be saved to /home/peng_yuxiang/tensorflow_tests/code/CPTN_Tensorflow/text-detection-ctpn/output/ctpn_end2end/voc_2007_trainval
Logs will be saved to /home/peng_yuxiang/tensorflow_tests/code/CPTN_Tensorflow/text-detection-ctpn/logs/ctpn/voc_2007_trainval/2017-10-26-16-03-04
/gpu:0
Tensor("data:0", shape=(?, ?, ?, 3), dtype=float32)
Tensor("conv5_3/conv5_3:0", shape=(?, ?, ?, 512), dtype=float32)
Tensor("rpn_conv/3x3/rpn_conv/3x3:0", shape=(?, ?, ?, 512), dtype=float32)
WARNING:tensorflow:<tensorflow.contrib.rnn.python.ops.core_rnn_cell_impl.BasicLSTMCell object at 0x7f0516b88450>: Using a concatenated state is slower and will soon be deprecated. Use state_is_tuple=True.
Tensor("lstm_o/Reshape:0", shape=(?, ?, ?, 128), dtype=float32)
Tensor("lstm_o/Reshape:0", shape=(?, ?, ?, 128), dtype=float32)
Tensor("rpn_cls_score/Reshape:0", shape=(?, ?, ?, 20), dtype=float32)
Tensor("gt_boxes:0", shape=(?, 5), dtype=float32)
Tensor("gt_ishard:0", shape=(?,), dtype=int32)
Tensor("dontcare_areas:0", shape=(?, 4), dtype=float32)
Tensor("im_info:0", shape=(?, 3), dtype=float32)
Tensor("rpn_cls_score/Reshape:0", shape=(?, ?, ?, 20), dtype=float32)
Tensor("rpn_cls_prob:0", shape=(?, ?, ?, ?), dtype=float32)
Tensor("Reshape_5:0", shape=(?, ?, ?, 20), dtype=float32)
Tensor("rpn_bbox_pred/Reshape:0", shape=(?, ?, ?, 40), dtype=float32)
Tensor("im_info:0", shape=(?, 3), dtype=float32)
Tensor("rpn_rois_data/Reshape:0", shape=(?, 5), dtype=float32)
Tensor("rpn_rois_data/PyFunc:1", dtype=float32)
Tensor("gt_boxes:0", shape=(?, 5), dtype=float32)
Tensor("gt_ishard:0", shape=(?,), dtype=int32)
Tensor("dontcare_areas:0", shape=(?, 4), dtype=float32)
W tensorflow/core/platform/cpu_feature_guard.cc:45] The TensorFlow library wasn't compiled to use SSE3 instructions, but these are available on your machine and could speed up CPU computations.
W tensorflow/core/platform/cpu_feature_guard.cc:45] The TensorFlow library wasn't compiled to use SSE4.1 instructions, but these are available on your machine and could speed up CPU computations.
W tensorflow/core/platform/cpu_feature_guard.cc:45] The TensorFlow library wasn't compiled to use SSE4.2 instructions, but these are available on your machine and could speed up CPU computations.
W tensorflow/core/platform/cpu_feature_guard.cc:45] The TensorFlow library wasn't compiled to use AVX instructions, but these are available on your machine and could speed up CPU computations.
Computing bounding-box regression targets...
bbox target means:
[[ 0. 0. 0. 0.]
[ 0. 0. 0. 0.]]
[ 0. 0. 0. 0.]
bbox target stdevs:
[[ 0.1 0.1 0.2 0.2]
[ 0.1 0.1 0.2 0.2]]
[ 0.1 0.1 0.2 0.2]
Normalizing targets
done
Solving...
/home/peng_yuxiang/tensorflow1.0/local/lib/python2.7/site-packages/tensorflow/python/ops/gradients_impl.py:91: UserWarning: Converting sparse IndexedSlices to a dense Tensor of unknown shape. This may consume a large amount of memory.
"Converting sparse IndexedSlices to a dense Tensor of unknown shape. "
Loading pretrained model weights from data/pretrain/VGG_imagenet.npy
assign pretrain model weights to conv5_1
assign pretrain model biases to conv5_1
ignore fc6
ignore fc6
assign pretrain model weights to conv5_3
assign pretrain model biases to conv5_3
ignore fc7
ignore fc7
ignore fc8
ignore fc8
assign pretrain model weights to conv5_2
assign pretrain model biases to conv5_2
assign pretrain model weights to conv4_1
assign pretrain model biases to conv4_1
assign pretrain model weights to conv4_2
assign pretrain model biases to conv4_2
assign pretrain model weights to conv4_3
assign pretrain model biases to conv4_3
assign pretrain model weights to conv3_3
assign pretrain model biases to conv3_3
assign pretrain model weights to conv3_2
assign pretrain model biases to conv3_2
assign pretrain model weights to conv3_1
assign pretrain model biases to conv3_1
assign pretrain model weights to conv1_1
assign pretrain model biases to conv1_1
assign pretrain model weights to conv1_2
assign pretrain model biases to conv1_2
assign pretrain model weights to conv2_2
assign pretrain model biases to conv2_2
assign pretrain model weights to conv2_1
assign pretrain model biases to conv2_1
iter: 0 / 180000, total loss: 1.8194, rpn_loss_cls: 0.6936, rpn_loss_box: 0.0755, rpn_loss: 1.0502, lr: 0.001000
speed: 18.390s / iter
image: img_5836.jpg iter: 10 / 180000, total loss: 1.6851, rpn_loss_cls: 0.6919, rpn_loss_box: 0.0470, rpn_loss: 0.9462, lr: 0.001000
speed: 20.903s / iter
image: img_2801.jpg iter: 20 / 180000, total loss: 2.7015, rpn_loss_cls: 0.6911, rpn_loss_box: 0.0254, rpn_loss: 1.9850, lr: 0.001000
speed: 18.728s / iter
image: img_1023.jpg iter: 30 / 180000, total loss: 1.9418, rpn_loss_cls: 0.6903, rpn_loss_box: 0.0254, rpn_loss: 1.2261, lr: 0.001000
speed: 14.129s / iter
image: img_3455.jpg iter: 40 / 180000, total loss: 1.7795, rpn_loss_cls: 0.6869, rpn_loss_box: 0.0630, rpn_loss: 1.0296, lr: 0.001000
speed: 16.138s / iter
image: img_1155.jpg
Traceback (most recent call last):
File "ctpn/train_net.py", line 39, in
restore=bool(int(0)))
File "/home/text-detection-ctpn/lib/fast_rcnn/train.py", line 338, in train_net
sw.train_model(sess, max_iters, restore=restore)
File "/home/text-detection-ctpn/lib/fast_rcnn/train.py", line 167, in train_model
sess.run(tf.assign(lr, lr.eval() * cfg.TRAIN.GAMMA))
File "/home/tensorflow1.0/local/lib/python2.7/site-packages/tensorflow/python/ops/gen_state_ops.py", line 47, in assign
use_locking=use_locking, name=name)
File "/home/tensorflow1.0/local/lib/python2.7/site-packages/tensorflow/python/framework/op_def_library.py", line 491, in apply_op
preferred_dtype=default_dtype)
File "/home/tensorflow1.0/local/lib/python2.7/site-packages/tensorflow/python/framework/ops.py", line 716, in internal_convert_to_tensor
ret = conversion_func(value, dtype=dtype, name=name, as_ref=as_ref)
File "/home/tensorflow1.0/local/lib/python2.7/site-packages/tensorflow/python/framework/constant_op.py", line 176, in _constant_tensor_conversion_function
return constant(v, dtype=dtype, name=name)
File "/home/tensorflow1.0/local/lib/python2.7/site-packages/tensorflow/python/framework/constant_op.py", line 169, in constant
attrs={"value": tensor_value, "dtype": dtype_value}, name=name).outputs[0]
File "/home/tensorflow1.0/local/lib/python2.7/site-packages/tensorflow/python/framework/ops.py", line 2354, in create_op
self._check_not_finalized()
File "/home/tensorflow1.0/local/lib/python2.7/site-packages/tensorflow/python/framework/ops.py", line 2077, in _check_not_finalized
raise RuntimeError("Graph is finalized and cannot be modified.")
RuntimeError: Graph is finalized and cannot be modified.
(1) Since I test is on CPU, I modify the text.yml -> STEPSIZE=50; for a few steps of training test.
(2) Model is load properly and the train loss changed, so the train step should be right;
But, when get the end, RuntimeError: Graph is finalized and cannot be modified.