Project of lightweight OCR contest.[url]

The best accuracy(score) is 0.7254, ranking 36th.

We tried different algorithms like CRNN, RARE, StarNet, and finally found CRNN is particularly useful in Chinese character recognition. Given this result, we adopted CRNN and focus on 1. obtaining a pruned (large-sparse) model from a large model and 2. designing a compact (small-dense) model.

Pruning is a common method to derive a large-sparse network, and it usually contains pre-training a model, pruning strategy, and fine-tuning. PaddleSlim supported L1-norm, L2-norm, and FPGM based pruning strategies. We also adopted EgaleEye, which proposed adaptive batch normalization to fast and accurately evaluate the network. (See work/prune.py)

We split the original training dataset into two subsets with split percentages of 80% training and 20% validation using random selection.

We use ResNet-18 as the backbone to trained a recognition model on the new training dataset normally from scratch as a baseline. After training, the large model achieved 0.7587 accuracy on the validation dataset.

Run work/prune.py to get the sensitivities of each layer.

After pruning 90% FLOPs and fine-tuning, we get a large-sparse model with an accuracy of 0.7177 and an allocated size of 8.9MB. Squeeze-and-Excitation(SE) can further improve the accuracy of the large-sparse model from 0.7177 to 0.7202 but increase allocated size to 9.8MB.

Overall, these results indicate that pruning a large model(ResNet18) to get an ultra-lightweight model is challenging. In our case, we pruned 90% FLOPs, causing accuracy to drop rapidly from 0.7587 to 0.7177. With an allocated size limited to 10MB, it is not easy to introduce more modules like SE into the network to improve accuracy further.

In recent years, there has been an increasing amount of literature on designing small-dense models for the optimal trade-off between accuracy and efficiency. PaddleOCR use MobileNetV3 as backbone to design ultra-lightweight model for Chinese character recognition. In this project, we train MobileNetV3-small-0.5 as our baseline with an accuracy of 0.6836.

SE module are integraed in a MobileNetV3. Some researchers proposed Coordinate Attention(CA) to replace SE for a better performance. We also try to replace SE module by CA module but the accuracy drop from 0.6836 to 0.6786. However, CA module reduces parameters and the allocated size of the model from 5.87MB to 4.49MB and from 6.9MB to 5.9MB, respectively.

MobileNetV3 introduces a new, fast, and quantization-friendly nonlinearity, h-swish function. Recently, TFNet proposed a novel activation function called ACON that explicitly learns to ACtivate the neurons Or Not. ACON-C function contains three learnable parameters p1, p2, and beta while MetaACON-C build a small network to learn beta. We try to replace HSwish and Relu with MetaACON-C, and we found that it is slower in back-propagation. The accuracy is 0.6787, and the allocated size is 10.5MB. Further experimental investigations are needed to estimate the practicality of MetaACON-C.

Inspired by Feature Pyramid Networks(FPN) and Dynamic Feature Pyramid Networks(DyFPN), we designed four FPNs to aggregate multi-scale feature information in recognition model.

We use to denote concatenation here. Given a list of input features with different scales , the output features are aggregated as

where denotes the level of pyramid. denotes convolution operation with different strides. denotes the resizing operation i.e. upsampling with a scale factor of (2,1).

Finally we sums the aggregated features to get output as:

In FPN-A, input features come from the first feature in state whereas in FPN-B, input features come from the last feature in state .

FPN-A achieved an accuracy of 0.7161 and allocated 4.8MB, whereas FPN-B achieved an accuracy of 0.7248 and allocated 7.5MB. The structures of FPN-A and FPN-B are shown below.

We redefine aggregated features and output as

Inspired by DyFPN, is added by more convolution operations with three different kernel size as:

Details of different models can refer to Table.

We use Adam optimizer with parameters beta1 of 0.9 and beta2 of 0.999 to train all models for 200 epochs, setting learning rate to 0.001, regularization parameter to 1e-5 and adpoting cosine learning rate schedule.

This project was undertaken to obtain a lightweight model from model pruning and compact model designing. One of the more significant findings to emerge from this project is that Feature Pyramid Networks(FPN) outputs aggregated features and helps to imrpove the recognition accuracy. Our final model reaches a training accuracy of 0.9670625 but a test accuracy of 0.735, which occurs overfitting. Therefore, the project was limited in several ways. First, the hyperparameters of training are not optimal. Adjust regularization parameter might alleviate overfitting. Second, the data is not preprocessed well. Data cleaning and data augmentation technology, e.g., removing dirty data, generating more data, can improve accuracy. Third, the neck and head of the model can be further optimized. Inspired by PlugNet, we also try to plug in a super-resolution unit to solve the low-quality text recognition problem but did not obtain a considerable result.

The overall directory structure of lightweight-OCR is introduced as follows:

lightweight-OCR   
├── 1833844.ipynb
├── data
│   ├── data87683
│   └── data87685
├── output
├── PaddleOCR
├── PaddleSlim
├── README.md
└── work
    ├── configs
    ├── label_dict.txt
    ├── label.txt
    ├── ppocr
    │   └── modeling
    │       ├── backbones
    │       └── necks
    └── tools
        ├── egaleeye_prune.py
        ├── export_pruned_model.py
        ├── infer
        ├── model_summary.py
        ├── prune.py
        └── train.py

• Python 3.7.10
• CUDA 10.1
• PaddleOCR-release/2.1
• PaddleSlim-release/2.0.0
• PaddlePaddle-2.0.2

Run command

python PaddleOCR/tools/train.py -c work/configs/rec_mobilev3_small_1_train

Run command

python PaddleOCR/tools/eval.py -c work/configs/rec_mobilev3_small_1_train -o Global.checkpoints=./output/rec_mobilev3_small_1.0/best_accuracy

Run command to export inference model

python PaddleOCR/tools/export.py -c work/config/rec_mobilev3_small_1_train.yml -o Global.checkpoints=./output/rec_mobilev3_small_1.0/best_accuracy Global.save_inference_dir=./output/rec_mobilev3_small_1.0/

inference model will be exported in output/rec_mobilev3_small_1.0/inference

Change 'rec_model_dir' and run command

python PaddleOCR/tools/infer/predict_rec.py --image_dir=./data/test_images/A榜测试数据集/TestAImages/ --rec_char_dict_path=./work/label_dict.txt --rec_model_dir=./output/rec_mobilev3_small_1.0/

result file will be saved in output/%Y-%m-%d-%H-%M-%S.log.