hsingjun / email-sentiment-classification Goto Github PK
View Code? Open in Web Editor NEWNatural Language Processing - Text Classification in Python
Natural Language Processing - Text Classification in Python
Tasks: 1. Spam/Ham classification on Enron email data 2. Sentiment classification on imdb data Comparing methods: 1. Naive Bayes 2. SVM 3. Maximum Entropy Model ———————————————————————————Summary of Python Scripts——————————————————— ——data preprocessing—— 1. enron_preprocess.py (train_PDF, test_PDF) Preprocessing enron data into Projct Data Format. The labeled data are transformed into the train_PDF file. The unlabeled testing data are transformed into the test_PDF file. Directories that contains spam files ('./enron1/spam', './enron2/spam', './enron4/spam', './enron5/spam'), ham files ('./enron1/ham', './enron2/ham', './enron4/ham', './enron5/ham') and test files ('./spam_or_ham_test') are hard coded in the script. Directories 'enron1', 'enron2', 'enron4', 'enron5' and 'spam_or_ham_test' should be placed in the same directory as the script 2. imdb_preprocess.py (raw_file, formatted_file) Preprocessing imdb data into Project Data Format. It can deal with both training data (with label) and testing data (without label). If there is label in the raw_file, it transforms the review score into POSITIVE/NEGATIVE label based the rule that if score>=7, label POSITIVE; if score<=4, label NEGATIVE. For both labeled and unlabeled data, increment the feature index by 1. 3. split_data.py (train_PDF, train, test, label, train_ratio) Split the whole labeled data into training set (with labels) and testing set (without labels). The actual labels of the testing set is stored in the 'label' file. The percentage of the training data among the whole set is specified by the parameter 'train_ratio) 4. preprocess_svm.py (train_PDF, train_SVM, test_PDF, test_SVM) Transform training and testing data in PDF into SVM required format. For training data, transform POSITIVE/NEGATIVE label to +1/-1. For test data, add +1 as an arbitrary label at the beginning of each line. 5. preprocess_mega.py (train_PDF, train_mega, test_PDF, test_mega) Transform training and testing data in PDF into MegaM required format. For training data, transform POSITIVE/NEGATIVE label to 1/0. For test data, add +1 as an arbitrary label at the beginning of each line. For both training and test data, replace the ':' between feature and value with a blank space. 6. trans_label.py (numeric_label, alphabetic_label, dataname) Transform numeric label (output from SVM-Light or MegaM) to alphabetic label: if numeric label > 0 -> POSITIVE/SPAM, else -> NEGATIVE/HAM. Parameter 'dataname' takes value 'imdb' or 'email'. If dataname = 'imdb', it converts numeric label to POSITIVE/NEGATIVE. If dataname = 'email', it converts numberic label to SPAM/HAM. ——Naive Bayes model training—— 1. nblearn.py (trainingfile, modelfile) Generate modelfIne from training file modelfile: 1st line: dataname ('imdb' or 'email') 2nd line: P(POSITIVE) 3rd line: P(NEGATIVE) 4th line: [P(feature(i)|POSITIVE)] 5th line: [P(feature(i)|NEGATIVE)] 2. nbclassify.py (modelfile, testfile) Do classification on test file using model file ——Result processing— 1. evaluation.py (true_label, predict_label) Compares the predict_label with true_label. Output the precision, recall, f1 score and accuracy. —————————————————————————————Summary of Commands——————————————————————— ———Data Preprocessing—— 1.Convert data into project format. Sentiment: $ python3 imdb_preprocess.py labeledBow.feat imdb_formatted $ python3 imdb_preprocess.py sentiment_test sentiment_test_formatted Spam: $ python3 enron_preprocess.py enron_formatted spam_test_formatted 2.Split data into training and testing groups. Sentiment: $ python3 split_data.py imdb_formatted imdb_trainT1 imdb_testT1 imdb_labelT1 0.75 $ python3 split_data.py imdb_formatted imdb_trainT2 imdb_testT2 imdb_labelT2 0.25 Spam: $ python3 split_data.py enron_formatted enron_trainT1 enron_testT1 enron_labelT1 0.75 $ python3 split_data.py enron_formatted enron_trainT2 enron_testT2 enron_labelT2 0.25 ———————75% training 25% testing——————— 1.Sentiment: a.Naive Bayes Command: $ python3 nblearn.py imdb_trainT1 imdb_nb_modelT1 $ python3 nbclassify.py imdb_nb_modelT1 imdb_testT1 > imdb_nb_predT1 $ python3 evaluation.py imdb_labelT1 imdb_nb_predT1 > imdb_nbT1f1 Result: precision(Pos) = 0.8714581893572909 precision(Neg) = 0.8256853396901073 recal(Pos) = 0.8117154811715481 recal(Neg) = 0.881641743557111 f1(Pos) = 0.8405265789035161 f1(Neg) = 0.8527465763963686 Accuracy = 0.84688 b.SVM-Light Command: $ python3 preprocess_svm.py imdb_trainT1 imdb_svm_trainT1 imdb_testT1 imdb_svm_testT1 $ ./svm_learn imdb_svm_trainT1 imdb_svm_modelT1 $ ./svm_classify imdb_svm_testT1 imdb_svm_modelT1 imdb_svm_outputT1 $ python3 trans_label.py imdb_svm_outputT1 imdb_svm_predT1 imdb $ python3 evaluation.py imdb_labelT1 imdb_svm_predT1 > imdb_svmT1f1 Result: precision(Pos) = 0.8601442458450925 precision(Neg) = 0.8810846128716106 recal(Pos) = 0.8828451882845189 recal(Neg) = 0.858097359210945 f1(Pos) = 0.8713468869123253 f1(Neg) = 0.8694390715667311 Accuracy = 0.8704 c.MegaM Command: $ python3 preprocess_mega.py imdb_trainT1 imdb_mega_trainT1 imdb_testT1 imdb_mega_testT1 $ ./megam_i686.opt -fvals binary imdb_mega_trainT1 > imdb_mega_modelT1 $ ./megam_i686.opt -fvals -predict imdb_mega_modelT1 binary imdb_mega_testT1 > imdb_mega_outputT1 $ python3 trans_label.py imdb_mega_outputT1 imdb_mega_predT1 imdb $ python3 evaluation.py imdb_labelT1 imdb_mega_predT1 > imdb_megaT1f1 Result: precision(Pos) = 0.8748793824380829 precision(Neg) = 0.8767908309455588 recal(Pos) = 0.8754425490827165 recal(Neg) = 0.8762328985046134 f1(Pos) = 0.8751608751608752 f1(Neg) = 0.8765117759388925 Accuracy = 0.87584 2.Spam: a.Naive Bayes Command: $ python3 nblearn.py enron_trainT1 enron_modelT1 $ python3 nbclassify.py enron_modelT1 enron_testT1 > enron_predT1 $ python3 evaluation.py enron_labelT1 enron_predT1 > enron_nbT1f1 Result: precision(Pos) = 0.9781021897810219 precision(Neg) = 0.9895287958115183 recal(Pos) = 0.9901477832512315 recal(Neg) = 0.9767441860465116 f1(Pos) = 0.9840881272949816 f1(Neg) = 0.9830949284785435 Accuracy = 0.9836065573770492 b.SVM-Light Command: $ python3 preprocess_svm.py enron_trainT1 enron_svm_trainT1 enron_testT1 enron_svm_testT1 $ ./svm_learn enron_svm_trainT1 enron_svm_modelT1 $ ./svm_classify enron_svm_testT1 enron_svm_modelT1 enron_svm_outputT1 $ python3 trans_label.py enron_svm_outputT1 enron_svm_predT1 email $ python3 evaluation.py enron_labelT1 enron_svm_predT1 > enron_svmT1f1 Result: precision(Pos) = 0.9325581395348838 precision(Neg) = 0.9885871704053523 recal(Pos) = 0.9897743300423131 recal(Neg) = 0.9252302025782688 f1(Pos) = 0.9603147451248717 f1(Neg) = 0.9558599695585996 Accuracy = 0.958205728697532 c.MegaM Command: $ python3 preprocess_mega.py enron_trainT1 enron_mega_trainT1 enron_testT1 enron_mega_testT1 $ ./megam_i686.opt -fvals binary enron_mega_trainT1 > enron_mega_modelT1 $ ./megam_i686.opt -fvals -predict enron_mega_modelT1 binary enron_mega_testT1 > enron_mega_outputT1 $ python3 trans_label.py enron_mega_outputT1 enron_mega_predT1 email $ python3 evaluation.py enron_labelT1 enron_mega_predT1 > enron_megaT1f1 Result: precision(Pos) = 0.9679531357684356 precision(Neg) = 0.9898074745186863 recal(Pos) = 0.9904795486600846 recal(Neg) = 0.9657458563535911 f1(Pos) = 0.9790867898222376 f1(Neg) = 0.9776286353467561 Accuracy = 0.9783822734642407 ———————25% training 75% testing——————— 1.Sentiment: a.Naive Bayes Command: $ python3 nblearn.py imdb_trainT2 imdb_nb_modelT2 $ python3 nbclassify.py imdb_nb_modelT2 imdb_testT2 > imdb_nb_predT2 $ python3 evaluation.py imdb_labelT2 imdb_nb_predT2 > imdb_nbT2f1 Result: precision(Pos) = 0.8590093160645308 precision(Neg) = 0.8202653799758746 recal(Pos) = 0.8087495988875816 recal(Neg) = 0.8679927667269439 f1(Pos) = 0.8331221420307422 f1(Neg) = 0.8434544420900304 Accuracy = 0.8384533333333334 b.SVM-Light Command: $ python3 preprocess_svm.py imdb_trainT2 imdb_svm_trainT2 imdb_testT2 imdb_svm_testT2 $ ./svm_learn imdb_svm_trainT2 imdb_svm_modelT2 $ ./svm_classify imdb_svm_testT2 imdb_svm_modelT2 imdb_svm_outputT2 $ python3 trans_label.py imdb_svm_outputT2 imdb_svm_predT2 imdb $ python3 evaluation.py imdb_labelT2 imdb_svm_predT2 > imdb_svmT2f1 Result: precision(Pos) = 0.8304547777210015 precision(Neg) = 0.8635805911879532 recal(Pos) = 0.8691838699326131 recal(Neg) = 0.8235294117647058 f1(Pos) = 0.849378070450507 f1(Neg) = 0.8430796036153761 Accuracy = 0.8462933333333333 c.MegaM Command: $ python3 preprocess_mega.py imdb_trainT2 imdb_mega_trainT2 imdb_testT2 imdb_mega_testT2 $ ./megam_i686.opt -fvals binary imdb_mega_trainT2 > imdb_mega_modelT2 $ ./megam_i686.opt -fvals -predict imdb_mega_modelT2 binary imdb_mega_testT2 > imdb_mega_outputT2 $ python3 trans_label.py imdb_mega_outputT2 imdb_mega_predT2 imdb $ python3 evaluation.py imdb_labelT2 imdb_mega_predT2 > imdb_megaT2f1 Result: precision(Pos) = 0.8431413178615831 precision(Neg) = 0.8668938228182018 recal(Pos) = 0.8704674296716226 recal(Neg) = 0.8389533028401234 f1(Pos) = 0.856586495447608 f1(Neg) = 0.85269474025623 ccuracy = 0.8546666666666667 2.Spam: a.Naive Bayes Command: $ python3 nblearn.py enron_trainT2 enron_modelT2 $ python3 nbclassify.py enron_modelT2 enron_testT2 > enron_predT2 $ python3 evaluation.py enron_labelT2 enron_predT2 > enron_nbT2f1 Result: precision(Pos) = 0.9782016348773842 precision(Neg) = 0.9822211397954214 recal(Pos) = 0.9826252528858741 recal(Neg) = 0.9776969696969697 f1(Pos) = 0.9804084540489194 f1(Neg) = 0.9799538330701008 Accuracy = 0.980183750675554 b.SVM-Light Command: $ python3 preprocess_svm.py enron_trainT2 enron_svm_trainT2 enron_testT2 enron_svm_testT2 $ ./svm_learn enron_svm_trainT2 enron_svm_modelT2 $ ./svm_classify enron_svm_testT2 enron_svm_modelT2 enron_svm_outputT2 $ python3 trans_label.py enron_svm_outputT2 enron_svm_predT2 email $ python3 evaluation.py enron_labelT2 enron_svm_predT2 > enron_svmT2f1 Result: precision(Pos) = 0.9095983364342782 precision(Neg) = 0.9877594465141033 recal(Pos) = 0.9890515292157562 recal(Neg) = 0.8998787878787878 f1(Pos) = 0.9476624857468643 f1(Neg) = 0.9417734365089432 Accuracy = 0.9448747973338137 c.MegaM Command: $ python3 preprocess_mega.py enron_trainT2 enron_mega_trainT2 enron_testT2 enron_mega_testT2 $ ./megam_i686.opt -fvals binary enron_mega_trainT2 > enron_mega_modelT2 $ ./megam_i686.opt -fvals -predict enron_mega_modelT2 binary enron_mega_testT2 > enron_mega_outputT2 $ python3 trans_label.py enron_mega_outputT2 enron_mega_predT2 email $ python3 evaluation.py enron_labelT2 enron_mega_predT2 > enron_megaT2f1 Result: precision(Pos) = 0.9703851106168793 precision(Neg) = 0.9860665844636252 recal(Pos) = 0.9865524217541354 recal(Neg) = 0.9693333333333334 f1(Pos) = 0.9784019827687949 f1(Neg) = 0.9776283618581907 Accuracy = 0.978021978021978
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.