An Empirical Study of Tokenization Strategies for Various Korean NLP Tasks [pdf]
Kyubyong Park*, Joohong Lee*, Seongbo Jang*, Dawoon Jung*
Accepted to AACL-IJCNLP 2020. (*indicates equal contribution)

Abstract: Typically, tokenization is the very first step in most text processing works. As a token serves as an atomic unit that embeds the contextual information of text, how to define a token plays a decisive role in the performance of a model.
Even though Byte Pair Encoding (BPE) has been considered thede facto standard tokenization method due to its simplicity and universality, it still remains unclear whether BPE works best across all languages and tasks. In this paper, we test several tokenization strategies in order to answer our primary research question, that is, "What is the best tokenization strategy for Korean NLP tasks?"
Experimental results demonstrate that a hybrid approach of morphological segmentation followed by BPE works best in Korean to/from English machine translation and natural language understanding tasks such as KorNLI, KorSTS, NSMC, and PAWS-X. As an exception, for KorQuAD, the Korean extension of SQuAD, BPE segmentation turns out to be the most effective.

pip install -r requirements.txt

There are 6 tokenization strategies for Korean. See here to prepare and use each strategy.

1. Consonant and Vowel
2. Syllable
3. Morpheme
4. Subword
5. Morpheme-aware Subword
6. Word

The corpus used for building vocabulary and training BPE models is as follows, which was extracted and refined via attardi/wikiextractor.

• Korean Wikipedia: https://dumps.wikimedia.org/kowiki
• English Wikipedia: https://dumps.wikimedia.org/enwiki

Recently, Korean-English parallel corpus was publicly released by AI Hub, which was gathered from various sources such as news, government web sites, legal documents, etc. We downloaded the news data which amount to 800K sentence pairs, and randomly split them into 784K (train), 8K (dev), and 8K (test).

We ran all the experiments using pytorch/fairseq (Ott et al., 2019), a PyTorch based deep learning library for sequence to sequence models.

fairseq-preprocess \
--source-lang ko \
--target-lang en \
--trainpref ./dataset/translation/mecab_sp-8k/train \
--validpref ./dataset/translation/mecab_sp-8k/dev \
--testpref ./dataset/translation/mecab_sp-8k/test \
--destdir ./dataset/translation/mecab_sp-8k/preprocessed/ko-en \
--srcdict ./resources/en_sp-32k/fairseq.vocab \
--tgtdict ./resources/mecab_sp-8k/fairseq.vocab

We used Transformer (Vaswani et al., 2017), the state-of-the-art model for neural machine translation. We mostly followed the base model configuration: 6 blocks of 512-2048 units with 8 attention heads.

fairseq-train ./dataset/translation/mecab_sp-8k/preprocessed/ko-en \
--arch transformer \
--share-decoder-input-output-embed \
--optimizer adam --adam-betas '(0.9, 0.98)' --clip-norm 0.0 \
--lr 5e-4 --lr-scheduler inverse_sqrt --warmup-updates 4000 \
--dropout 0.3 --weight-decay 0.0001 \
--criterion label_smoothed_cross_entropy --label-smoothing 0.1 \
--max-epoch 50 \
--batch-size 128 \
--save-dir translation_ckpt/mecab_sp-8k/ko-en \
--disable-validation

We report BLEU scores on both the dev and the test sets using Moses multi-bleu.perl script. Following WAT 2019 (Nakazawa et al., 2019), Moses tokenizer and MeCab-ko are used for tokenizing the evaluation data.

fairseq-generate ./dataset/translation/mecab_sp-8k/preprocessed \
--path translation_ckpt/mecab_sp-8k/checkpoint_best.pt \
--batch-size 512 \
--beam 5 \
--remove-bpe sentencepiece

For each tokenization strategy, pre-training of BERT-Base model (Devlin et al., 2019) was performed with a Cloud TPU v3-8 for 1M steps using the official code of google-research/bert.

We set the training hyper-parameters of all models as follows: batch_size=1024, max_sequence_length=128, learning_rate=5e-5, warm_up_steps=10000.

Because the Korean Wiki corpus (640 MB) is not enough in volume for the pre-training purpose, we additionally downloaded the recent dump of Namuwiki (5.5 GB) and extracted plain texts using Namu Wiki Extractor.

After converting each pre-trained model in TensorFlow into PyTorch, we fine-tuned them using huggingface/transformers (Wolf et al., 2019).

example:

python tasks/<TASK_NAME>/run_train.py --tokenizer <TOKENIZER_NAME>

@article{park2020empirical,
  title={An Empirical Study of Tokenization Strategies for Various Korean NLP Tasks},
  author={Park, Kyubyong and Lee, Joohong and Jang, Seongbo and Jung, Dawoon},
  journal={arXiv preprint arXiv:2010.02534},
  year={2020}
}

For pre-training models, we used Cloud TPUs provided by TensorFlow Research Cloud program.