carrielui / pathpredictionfortextclassification Goto Github PK

Source code for WWW 2019 paper "Efficient Path Prediction for Semi-Supervised and Weakly Supervised Hierarchical Text Classification"

Readers are welcomed to fork this repository to reproduce the experiments and follow our work. Please kindly cite our paper.

@article{xiao2019efficient,
title={Efficient Path Prediction for Semi-Supervised and Weakly Supervised Hierarchical Text Classification},
author={Xiao, Huiru and Liu, Xin and Song, Yangqiu},
journal={arXiv preprint arXiv:1902.09347},
year={2019}
}

• numpy
• networkx
• scipy
• sklearn
• nltk

Filter data with multilabels.

python filter_multilabels.py

Split data to train set and test set.

python split_data.py

Thirdly, generate data managers.

python build_data_managers.py

If you need to generate files in svmlight/libsvm file format, you can run generate_svmlight_format.py.

python generate_svmlight_format.py

If you need to run LIBLINEAR, you need to download it and put it in this folder, or create a soft link.

You can run NB_EM.py, LR_SVM.py, HierCost.py, and LIBLINEAR.py to train different models.

python NB_EM.py
python LR_SVM.py
python HierCost.py
python LIBLINEAR.py

You can get the results in data/20ng/0.1 when settings.label_ratios include 0.1 and settings.data_dirs include data/20ng.

Labeled

Dataless

Naive Bayes and the EM algorithm do not have many parameters, but many parameters can affect the data distribution. Most parameters are defined in settings.py

• INF: the initial value of log-likelihood
• EPS: the value to avoid division by zero
• max_vocab_size: the maximum size of vocabulary (Smaller values can speed algorithms up)
• train_ratio: the ratio of training size to test size
• label_ratios: the ratios of labeled size to unlabeled size
• times: times that we split data in the same setting in order to the reduce random error
• main_metric: the metric used to stop in EM algorithms
• soft_sim: using similarities on all classes or the maximum similarity

In some NB_EM baselines, soft_pathscore=True means we don't provide the hierachical information to classifiers but models can learn this information by themselves.