So we can train, test, then continue re-training.

Some possible questions:

• Does it feel like we're making good progress/that we're focused enough?
• Suggestions on more things we can analyze? Right now, we're sort of exploring a bunch of things without necessarily a clear endgoal?

Do we also have any technical questions we can ask?

Might require some refactoring.

Approach as reading comprehension problem?

Implement code to save and load trained model. Probably dependent on #3

• Bag-of-words model
• Logistic regression model
• RNN model

Tasks 2 and 3 can probably be combined, since they both use tensorflow?

• Make account + project on Azure
• Email Hannah account email to get credits
• Provision instance and install CUDA + other Nvidia stuff + Tensorflow

This probably isn't super crucial -- I think we both have access to GPUs atm, so we can afford to put off at least the installation phase for a few days, since that's tedious and finicky.

For the sake of due diligence, we should check if the annotators for the wikipedia dataset are noticeably biased in some way.

This data may already be in the wikipedia paper; we may need to collect it ourselves.

We should probably make sure to have a short blurb about this in our final report.

This is more of an engineering concern, but if this were to become an actual product, it'd probably be useful to make sure the classifier is performant/can handle a reasonable load while running on a variety of different hardware.

It might be worth conducting experiments/explicitly trying to optimize the speed of our classifiers/other tensorflow stuff?

Currently, the RNN model is hovering around 0.8 to 0.83 AUC score despite doing a bunch of tweaking. Things to try:

• Scramble data before each epoch
• Try porting RNN to a character RNN
• Try using a static RNN instead of a bidirectional RNN
• Vary different parameters by hand
• Try stacking multiple RNNs
• Try using variants of different dynamic RNNs
• Try using an actual grid search thing. It'll probably take hours to run, but whatever.
• Try using optimizers other then Adam (SGD? Momentum? AdaDelta?)
• Try varying learning rate
• Dig into misclassified results to get better insights into what the problem is

Not a huge priority, but it'd be cool to add tooling that...

• Checks for PEP 8 compliance
• Runs mypy (to typecheck code)

We should also:

• Document usage in readme

This web interface should...

• Accept arbitrary text as input
• Report the output of our models on different metrics
• Make sure we can run multiple tensorflow models

It'd be cool if we can have something basic working for our Monday demo, then add more features later for our final demo.

• Load wikipedia toxicity dataset
• Load wikipedia aggressiveness dataset
• Figure out if all of the comments in all datasets are the "same", and see if we can't store them all in the 'comment' object.
• Cache loaded data from each dataset

Treat as machine translation problem?

It'd be cool if we could have some sort of slider to scale the level of "aggressiveness/toxicity"?

TODO:

• Research approach
• Add more TODOs

We'll probably follow approximately the same interface as the character n-gram model.

• Refactor RNN model
• Refactor bag-of-words model
• Refactor character n-gram model
• Make base class or abc to help code typecheck
• Make sure all models can accept multiple classes?
• Strip out all dataset-specific code

Ideas to explore:

• Apply a mask before computing loss
• Look into using dynamically-resizing RNNs, and having the width vary per batch

Potential issues:

• Comment width can vary widely; how do we handle outliers?

The command line tool should let us...

• Pick which model we want to use
• Pick which dataset we want to analyze
• Provide model-specific options
• Choose between re-training or using an already-trained model
• Lets us chose if we want to clear any caches/logs or reuse them (for tensorflow)
• Lets us chose the output location of any data/logs

Probably dependent on #3.