parkerburchett / naturallanguage Goto Github PK

Look at BetterTrainingData.py and rewrite that to pickle all the things that take forever to load.

You could do a blog post about this. In theory, smaller batches means a more noisy gradient. This would make it more generalizable but have a higher initial time cost.

apparently this works well in bag of words text classification.

Right now you have

classifierList = open("C:/Users/parke/Documents/GitHub/NaturalLanguage/NaturalLanguage/OwnPrograms/pickled_TrainedClassifierList.pickle", "rb")
classifiers = pickle.load(classifierList)
classifierList.close()

testingSetIn = open("C:/Users/parke/Documents/GitHub/NaturalLanguage/NaturalLanguage/OwnPrograms/pickled_TestingData.pickle", "rb")
TestingSet = pickle.load(testingSetIn)
testingSetIn.close()

word_featuresIN = open("C:/Users/parke/Documents/GitHub/NaturalLanguage/NaturalLanguage/OwnPrograms/pickled_word_features.pickle", "rb")
word_features = pickle.load(word_featuresIN)
word_featuresIN.close()

documentsIN = open("C:/Users/parke/Documents/GitHub/NaturalLanguage/NaturalLanguage/OwnPrograms/pickled_documents.pickle","rb")
documents = pickle.load(documentsIN)
documentsIN.close()

You should put that in a method in general so that it is more clear

Look at this for inspiration: https://www.kaggle.com/paoloripamonti/twitter-sentiment-analysis

I don't know why this classifier takes forever but it is not loading. Replace this classifier with a different classifier.

You want to have 7 different classifiers so that there is an odd number in the vote classifier

Use this to get the rates of positive and negative.

Need to test get results for testing on a random 1000 tweets from the twitter API. See what portion are positive and what portion are negative and what portion are posititve. .

Right now this is every method. You should break them up into separate modules so that it is more compartmentalized.

This will solve the memory

This method needs to do only one thing and one thing well. Make this part of a wider refactor of general.py

Based on differnt params and train test split you need to write something to compare. look at the README.txt in Custom_NLTK_Utils

Remove \n and emoticons at least.

Look at POS3.

The word features are all the same. actually Identical. when they should be different based on the rt of speech. But the accuracy is different for the classifiers.

I have no idea why this is the case. The differences might be just random variation.

Need to rewrite this to send all the pickled info into a single folder, and write a method that will unpickle all the stuff in that folder as is.

This would limit a lot of the noise in the classifiers.

Look at partsOfsSeech 3.txt

In the old determine Ideal Algo Params you had hardcoded every algo to use 3000 most frequent words

Right now there is an error deep in numpy. I should make sure that the pickled algos work and are accurate, before I try to make sentiment work.

Idea of how to solve:
Add method to VoteClassifier that will return an what each component algo one votes for.

Get 10k raw tweets from twitter and run the VoteClassifier on them. You can then put it in excel to determine the correlation between them. Add this to your blog.

I think it might because you feed in the information straight from documents and you don't limit features from documents. Need to verify

See the kaggle dataset in "Datasets"

Pseudocode:
You need to have word_features here as well.

Convert Tweet via word_tokenize -> call this 'words'

Convert 'words' to a long boolean vector -> call this tweet_as_vector

Pass 'tweet_as_vector' to the VoteClassifier -> return positive or negative

You might want to create 9 different slight variations of the classifier (eg different cost functions or initial parameters). Then when there is 7,8 or 9 votes for one of the modes return 'Positive" or "Negative" or "Unsure"

Right now a lot of words have weight that should not.

Right now it does all of the steps between raw data and generating features sets.

These should all be in their own methods

I expect training on the entire dataset will take about an hour. s

documents
all_words
word_features
feature_sets
training_Data
testing_Data

A List of Trained Algos