wpm / mycroft Goto Github PK

Specify as a train argument.

The command line assumes it's reading model directory, not a model file.

Check to see whether the argument is reading a file or directory then load accordingly.

Go over the codebase and insert inline documentation for the classes and functions.

Say "process" text not "parse", maybe brief justification for why I'm using spaCy instead of Keras' built-in tokenizer.

If you don't specify a validation set, save best model does nothing. Need to explicitly save the model in this case.

The "coarse" labels In the Stanford Treebank set include a "neutral" value which should not be included in any evaluation. It would be good to be able to handle this directly from the Mycroft command line.

Add --label-name and --omit-labels just as in the evaluate command.

If you supply these labels we can also print the accuracy to standard out.

It would be waste of time for me to try and put all the possible Keras options you could tweak into the command line interface. Ultimately Mycroft is more useful if you can write your own models in code.

Still Mycroft provides some advantages over starting from scratch every time: e.g. all the pandas data processing, seamless incorporation of word embedding vectors, the prediction and evaluation code which remains the same regardless of model, and a command line interface. It would be good to be able to easily leverage that for new models.

Do this by adding command line support to the TextEmbeddingClassifier. Add to the interface a command_line function that returns an argparse subparser. Refactor the current models to use this and call it from console.py:main to build the command line parser. Also export a main function that somebody else's code could call passing in a TextEmbeddingClassifier as an argument. This would build a command-line interface for just this class.

Would need to explain this clearly in the README, where the value-add over writing Keras code from scratch lies.

Mycroft is primarily a tool for doing learning over word embeddings, so this is here as a convenient baseline.

There's no point in putting a lot of work into supporting all the various grid parameters of an SVM, or different learning algorithms. Anyone who wants to seriously explore this space can write their own sklearn code.

Keras supports logging to the TensorBoard format. Provide a command line option to use this. Maybe settle on TensorBoard as Mycroft's preferred format for detailed analysis of the training process. Maybe even ditch the history.json file.

The demo runs out of memory. The machine freezes. I'm doing something wrong.

Before trying to reproduce published results #11 just get a set of results for the Stanford Sentiment data set. This is a smoke test to give me some confidence that this is working for release 1.0.0.

Do this an experimental grid for a few different model/hyperparameter combinations on the coarse (either "positive" or "negative") sentiment samples. Do an 80/20 split and measure validation accuracy and loss.

These were run on a G1 GPU server on Floydhub with 4 cores, 1 GPU, 61G memory, and 12G GPU memory.

Conclusions:

• SVM and sequential models are about the same. Both outperform a neural bag of words model.
• SVM trains a lot faster.
• The literature says 80% is a decent baseline for this data set, so the code does appear to be working.

Instead of creating a separate training command for each model, make models a subcommand of a shared train command. The training arguments will be the same across all of them.

The train command could then also have a load subcommand that reloads an existing model instead of creating a new one.

Have a demo command that given a text file randomly shuffles the words in some of its lines then builds and runs a classifier to detect the difference between shuffled and non-shuffled English text.

It writes the data files and output model to the current directory so you have an example of how to configure your own data files.

If no vocabulary size argument is specified simply make the vocabulary the set of all types in the language model for which we have embedding vectors. Don't tie this to the number of types in the data.

Including all the types in the language model vocabulary will always be the "best" we can do, so it should be the default.

Allow the model to have multiple layers. You could input a list of hidden unit sizes.

The standard convolutional model generates features from windows of contiguous words. If we are given a syntactic parse of a sentence we can instead generate a window for each syntactic component. Perhaps the pooling layer could extract the maximum value from sets of constituents at the same parsing depth.

The more general idea is to redefine the notion of "window" to be something more relevant than contiguous tokens. Maybe implement a more generic version of Keras' Conv1D.

Check the literature to see if this has been done. This is basically a research paper idea.

Easier to do with the change in #36.

The default is still "en", but you should be able to specify something else on the command line.

Implement autoencoding strategy described in

Gan, Zhe, Pu, Yunchen, Henao, Ricardo, Li, Chunyuan, He, Xiaodong, and Carin, Lawrence. Unsupervised learning of sentence representations using convolutional neural networks. arXiv preprint arXiv:1611.07897, 2016.

For any model you should be able to run autoencoding, the save the weights.

Separate code that interacts with the console or the file system from code that does the work. All the former should go into console.py. The latter should be organized as an API.

(One exception: training will write a model file.)

Add a command-line option to specify the embeddings as learnable during training.

1. Add learning rate as a parameter since it is common to all optimizers.
2. Optionally use a reduce learning rate on plateau callback.
3. Optionally use an early stopping callback.

These may help eke out a few more good epochs from training.

It's too easy to break this when you're futzing around with the command line code.

Optionally weight input vectors by TF-IDF. Should be easy to get from scikit-learn.

Use all the standard github tools. Have autogenerated API documentation, tutorials on writing your own classifiers, writeups of experiments comparable to published baselines.

For example

mycroft train conv ../data/MR/scaledata.csv --validation-fraction 0.1 --sequence-length 100 --text-name subj --label-name label.3class
usage: mycroft train conv [-h] [--limit LIMIT]
                          [--validation-fraction FRACTION]
                          [--validation-data FILE] [--text-name NAME]
                          [--label-name NAME]
                          [--omit-labels [LABEL [LABEL ...]]]
                          [--epochs EPOCHS] [--batch-size SIZE]
                          [--model-directory DIRECTORY]
                          [--logging {none,progress,epoch}]
                          [--tensor-board DIRECTORY]
                          [--sequence-length LENGTH] [--vocabulary-size SIZE]
                          [--dropout DROPOUT] [--filters FILTERS]
                          [--kernel-size SIZE] [--pool-factor FACTOR]
                          [--language-model MODEL]
                          TRAINING-FILE
mycroft train conv: error: argument --sequence-length: invalid NoneType value: '100'

Don't serialize the vocabulary when serializing an TextSequenceEmbedder object. Instead, recreate the vocabulary when deserializing.

This would require breaking off the vocabulary creation at the end of the TextSequenceEmbedder constructor into its own class and overriding the __getstate__ and __setstate__ pickle functions.

This is good because you may want to have huge vocabularies (e.g. 2,000,000 words) that work just fine as embedding matrices but cause pickle to run out of memory when trying to serialize them.

Complete when I have 100% code coverage or near to it.

These two arguments would be mutually exclusive.

Want this for the Stanford Sentiment Treebank because that experimental setup specifies train, dev, and test sets.

Mention the demo command.

Point out the classes that comprise the programmatic interface.

Include numbers generated by #20.

This could be helpful for data augmentation experiments.

That way I can mention the pip install line in the README.

Here's a tutorial.

Give callers the option of specifying l₁ or l₂ and a regularization rate.

Apply this to the Dense layer at the end of the two built-in models.

Again, don't want to add too many more bells and whistles to the built-in models beyond this.

As an alternative to specifying a validation set (#31) or proportion, you should be able to specify the number of folds for cross validation. Models will be saved to subdirectories of the model directory. Print a mean and standard deviation of the statistics.

Obviously you can do this by making multiple runs with a validation proportion, but that's a lot of command lines to keep track of.

Optionally allow you to predict and evaluate a test file at the end of training.

Saves time running a separate predict job.

Change the embedding code to take the mean of the token vectors instead of putting them in a matrix.

I put that in to make things easier for Anna's intern, but really supporting every possible text classifier is beyond the scope of this project. Mycroft should just focus on deep learning methods. If you want to do something else, you're better off using scikit-learn.

Also SVM support complicates the architecture and user interface somewhat.

Instead of specifying the vocabulary size and creating an Embedder from that, do a pass over the training data and build and embedder that has precisely all the words in that vocabulary.

Actually, the interesting experimental question is: does this help, or is it just insignificant fine tuning?

Don't try to manage part of this outside the models in the pandas frames. It'll be more robust to put it all in one place.

Make sure I can install and run unit tests in a Python 2 environment.

Optionally write the loss scores to a .JSON file in case you want to plot them later.

@wpm commented on Wed Aug 02 2017

I don't pass mask_zero = True into the embedder for the ConvolutionNetClassifier even though I do for the RNNClassifier. Should I?

Mycroft should support multiple text inputs so that it can be used for multi-text labeling tasks like textual entailment.

TextEmbeddingClassifier.train would take a list of texts arguments and pass a list of training_vectors to self.model.fit. (Maybe the list would be converted to a single vector if there is only one text input.)

The --text-name option would take multiple arguments. I'd need to add a mechanism for the TextEmbeddingClassifier subclass to specify what these would be. The default would remain a single argument called "text".

This is a nice-to-have. Of course you can always do grid search by generating Mycroft command lines outside this package, but maybe it's worth doing inside this code if (a) there would be a convenient command line interface (b) it doesn't require much implementation work.

The criteria for (b) is "can I wrap these models in Keras' scikit-learn layer"? If not, I'm not going to reimplement scikit-learn's grid search capability.

For (a) I guess you could simply change all the command line arguments to take multiple values, though you might end up wanting to be able to launch a job from a configuration file.

This would need some investigation to see if it's worthwhile.

This may be necessary for large data sets.

Just a stringification of the model. Remove the details command.

Explain that acc is classification accuracy and loss is cross-entropy loss in the documentation and the online help.

Use this code to reproduce published results for RNNs for text classification. Put the results and references in the README.

Nothing exhaustive, just reproduce something to show that it's working.

Another kind of neural model for sequences.