Training deep learning models to classify disaster-related tweets in the context of crisis management. Published as "Using deep learning and social network analysis to understand and manage extreme flooding" in Journal of Contingencies and Crisis Management
Is your feature request related to a problem? Please describe.
Social media text has a lot of non-standard English, leading to a lot of out-of-vocabulary words.
Subword tokenization algorithms such as byte-pair encoding can help with this issue, as well as bypass the need for text normalization which can lose a lot of information.
Describe the solution you'd like
Implement subword tokenization through AllenNLP library and evaluate it's impact on tweet classification.
Describe alternatives you've considered
Word-level tokenization + normalization: not ideal due to reasons outlined above.