Improved CRNN,ASTER,DAN on different text domains like scene text, hand written, document, chinese/english, even ancient books
Update🙂🙂
Date
Description
7/30
Checkpoint for CRNN on IAM dataset has been released. You can test your English handwritten now
7/31
Checkpoint for CRNN on CASIA-HWDB2.x has been released. You can test your Chinese handwritten now
8/3
New Algorithms! ASTER is reimplemented here and checkpoint for scene text recognition is released
8/5
Checkpoint for ASTER on IAM dataset has beem released. It's much more accurate than CRNN due to attention model's implicit semantic information. You should not miss it😃
8/8
New Algorithms! DAN(Decoupled attention network) is reimplented. checkpoint forb both scene text and iam dataset are realesed
8/11
New Algorithms! ACE(Aggratation Cross-Entropy). It's a new loss function to handle text recognition task. Like CTC and Attention
8/17
Retrained ACE and DAN; Add a powerful augmentation tool
9/7
Training SRN and so on.
1. Welcome!😃😃
Now I'm focusing on a project to build a general ocr systems which can recognize different text domains. From scene text, hand written, document, chinese, english to even ancient books like confucian classics. So far I don't have a clear idea about how to do it, but let's just do it step by step. This repository is suitable for greens who are interesed in text recognition(I am a green too😂).
639 test images instances. It is specifically designed to evaluate perspective distorted textrecognition. It is built based on the original SVT dataset by selecting the images at the sameaddress on Google Street View but with different view angles. Therefore, most text instancesare heavily distorted by the non-frontal view angle.
2077 test images instances. As text images were taken by Google Glasses without ensuringthe image quality, most of the text is very small, blurred, and multi-oriented
IAM dataset is based on handwritten English text copied from the LOB corpus. It contains 747 documents(6,482 lines) in the training set, 116 documents (976 lines)in the validation set and 336 documents (2,915 lines) in the testing set
I reimplemented the most classic and wildly deployed algorithm CRNN. The orignal backbone is replaced by a modifyied ResNet and the results below are trained on MJ + ST.
#
IIIT5K
SVT
IC03
IC13
IC15
SVTP
CUTE
CRNN(reimplemented)
91.2
84.4
90.8
88.0
73.1
71.8
77.4
CRNN(original)
78.2
80.8
89.4
86.7
-
-
-
Some recognion results
Image
GT
Prediction
I am so sorry
'iamsosory'
I still love you
'istilloveyou'
Can we begin again
'canwebeginagain'
note that we only predict 0-9, a-z. No upper case and punctuations. If you want to predict them, you can modify the code
4.1.2 On Handwritten
Relative experiments are conducted on IAM dataset and CASIA-HWDB
Dataset
Word Accuracy
IAM(line level)
67.2
CASIA-HWDB2.0-2.2
88.6
Some recognion results
Image
GT
Prediction
Just Somebody I Can Kiss
'Just Somebody I can kiss'
Just something I can turn to
'Just something I can turn to'
昨夜西风凋碧树,独上西楼,望尽天涯路。
'昨夜西风调瑟树,独上西楼。望尽天涯路'
衣带渐宽终不悔,为伊消得人憔悴
'衣带渐宽终不海,为伸消得人憔悴'
众里寻他千百度,蓦然回首,那人却在灯火阑珊处
'众里寻他千百度,暮然回首,那人却在灯火闻班然'
你好,**
'你好,**'
欢迎来到重庆
'欢迎来到重庆'
Chinese handwritten are sufferd from imbalanced words contribution. So sometimes it's hard to recognize some rare words
4.2 ASTER
4.2.1 On Scene Text
ASTER is a classic text recognition algorithms with a TPS rectification network and attention decoder.
#
IIIT5K
SVT
IC03
IC13
IC15
SVTP
CUTE
ASTER(reimplemented)
92.9
88.1
91.2
88.6
75.9
78.3
78.5
ASTER(original)
91.93
88.76
93.49
89.75
#
74.11
73.26
Some recognion results
Image and Rectified Image
GT
Prediction
COLLEGE
'COLLEGE'
FOOTBALL
'FOOTBALL'
BURTON
'BURTON'
4.2.2 On Handwritten
Relative experiments are conducted on IAM dataset and CASIA-HWDB
Dataset
Word Accuracy
IAM(line level)
69.8
CASIA-HWDB2.0-2.2
The model fails to convergence and I am still training
Some recognion results
Image
GT
Prediction
Coldplay is my favorate band
'Coldplay is my favorate band'
Night gathers and now my watch begins
'Night gathers and now my watch begins'
You konw nothing John Snow
'You konw nothing John snow'
DAN
4.3.1 On Scene Text
#
IIIT5K
SVT
IC03
IC13
IC15
SVTP
CUTE
DAN1D(reimplemented)
91.2
83.8
89.4
88.7
72.1
70.2
74.7
DAN1D(original)
93.3
88.4
95.2
94.2
71.8
76.8
80.6
4.3.2 On Handwritten
Relative experiments are conducted on IAM dataset and CASIA-HWDB
Dataset
Word Accuracy
IAM(line level)
74.0
CASIA-HWDB2.0-2.2
Some recognion results
Image
Prediction
'I have seen things you people would not believe lift'
'Attack ships on fire off the shoulder of Orien'
'I have watch bearans gitter in the does near the Tarhouser'
'All those moments will be lost in time'
'like tears in the rain'
ACE
4.4.1 On Scene Text
ACE is simple yet effective loss funciton. However, there is still a huge gap with CTC and Attention
#
IIIT5K
SVT
IC03
IC13
IC15
SVTP
CUTE
ACE(reimplemented)
84.8
76.7
84.0
82.6
65.3
64.8
68.8
ACE(original)
82.3
82.6
92.1
89.7
#
#
#
How to use
It's easy to start the training process. Firstly you need to download the datasets
required.
.py", line 1045, in load_state_dict
self.class.name, "\n\t".join(error_msgs)))
RuntimeError: Error(s) in loading state_dict for ModelBuilder:
size mismatch for decoder.weight: copying a param with shape torch.Size([6018, 512]) from checkpoint, the shape in current model is torch.Size([12, 512]).
size mismatch for decoder.bias: copying a param with shape torch.Size([6018]) from checkpoint, the shape in current model is torch.Size([12]).
我是使用的您的ResNet_CRNN_IAM模型进行训练手写汉字的数据集,但是将模型导出为onnx的时候遇到了问题,我看您在网络中定义了forward和inference(您的inferrence应该是笔误)两个方法,我给的输入dummy_input 是torch.randn(1, 3, 192, 2048)的时候,会报
x, rec_targets, rec_lengths = input_dict['images'],
IndexError: too many indices for tensor of dimension 4,这应该跟您在forward中定义需要读入input_dict 的字典作为输入参数有关,但我将dummy_input 换成这个字典的参数,还是无法正常生成。请问我应该怎么修改呢