We introduce an extra visual transformer as the alignment-ware image encoder and an extra text transformer as the alignment ware text encoder before multimodal fusion. We consider alignment in the following three aspects: 1) document-level alignment by leveraging the cross-modal and intra-modal contrastive loss; 2) global-local alignment for modeling localized and structural information in document images; and 3) local-level alignment for more accurate patch-level information. For more details, please refer to our paper: Alignment-Enriched Tuning for Patch-Level Pre-trained Document Image Models[pdf]

git clone https://github.com/MAEHCM/AET.git && cd AET

First set up the required dataset under the appropriate path ，Select the corresponding model in { }，funsd,cord or cdip

cd preprocess
python preprocess_{}.py

Then execute the corresponding finetune file to fine tune the model ，Select the corresponding model in { }，funsd,cord or cdip

python -m torch.distributed.launch --nproc_per_node=2 --use_env python finetune_{}.py --config Pretrain.yaml --output_dir

We referenced the code of ALBEF and TCL when implementing AET in github, In this repository, we used three public benchmark datasets, FUNSD , CORD , RVL-CDIP.

If you find this repo is useful, please consider to cite our paper:

TBD