Original paper: Towards better substitution-based word sense induction - https://arxiv.org/abs/1905.12598
Adapted for Reddit data: https://github.com/lucy3/ingroup_lang
Python 3.7
Install requirements.txt with pip -r
This will install python pacakges including pytorch and huggingface's BERT port.
(for CUDA support first install pytorch accroding to their instructions).
Run download_resources.sh to download datasets.
Run wsi_bert.py for sense induction on both SemEval 2010 and 2013 WSI task datasets.
Logs should be printed to "debug" dir.
- train_semeval.py: training for SemEval tasks
- reddit_prep.py: formats Reddit data
- wsi_reddit.py: train / learn clusters on Reddit examples
- match_reddit.py: matches Reddit examples to clusters obtained after training
- inspect_senses.py: examines resulting senses
- scheduler.py: for submitting jobs
- wsi/wsi_clustering.py: includes modifications that save learned clusters and a matching process for new examples