nlesc / mcfly Goto Github PK

Prepare dataset EEG Utrecht

Investigate what ideal shape of data input should be for Keras in the context of time series

see #34

Investigate and decide whether to implement DTW

Implement per channel CNN in normal CNN architecture, without LSTM. Add this to architecture generation functions.

Document what choices we made.
Model types, achitectures, default parameters etc.

setup gh-pages without content

Read-up on Keras and continue literature review on time series

possible parameter ranges to be set etc.

Rebuild the model architectures in modelgen.py for hyperas, and run them on a small dataset.

Prepare dataset London: data format + take subset of two easy to recognize classes + standardize window length, e.g. 10 minutes

I am listing some research ideas we could look into after there is a first prototype. These may need to be split into seperate issues but for now it may be easier to group them:

1. Is processing a multi-channel with nbcol = 1 the same as splitting up the data and processing them in separated sequences and the connecting the branches in a fully connected final layer? The code will be less compact in the latter case, but what is the impact on classification performance?
2. Should dropout be applied in between convolutional layers and/or in between LSTM layers? What do other do and what is the impact on the performance?
3. L2 regularisation is not mentioned in the article by Ordonez, is L2 regularization the unmentioned standard for CNN?
4. What is the impact of return_sequence = True versus False in the LSTM layer?
5. How many dense layers do we need to have equal classification performance as one LSTM layer? 6. How many convolutional layers do we need to have equal classification performance as three convolutional and three LSTM layers?

Build simple classifier for London data

Prepare small dataset UCR time series classification archive:

http://www.cs.ucr.edu/~eamonn/time_series_data/
The password for the data (link to 350 Mb zip) is: attempttoclassify
http://www.cs.ucr.edu/~eamonn/time_series_data/UCR_TS_Archive_2015.zip

Use a (cookiecutter) template to set up the repository structure according to the eStep guidelines.

Right now there is one uniform parameter search for e.g. learning rate.
It would be nice if there's an automatic search strategy from course to finegrained.

Add regularization as parameter to optimize in modelgen.py.

Build first classifier in Keras, locally (no DAS5): programming + training + evaluating using as a refernece: published UCR results or use shallow classifier if possible

https://github.com/maxpumperla/hyperas

Create architecture specific for one datasets (UCR) and stored the model objects

Input:

• Data
• Non-data dependent hyper-parameters: Number of nodes in layer, learning rate, convolution size,...

Output:

• Compile model object

Gull dataset

Extend the functionality from issue #19 to incorporate architectures that are like those presented in the paper of Ordonez et al, 2016.

Find out how to transfer the final hidden state for some sample on to the first hidden state for the next sample. This was done in Plotz (see literature on onedrive).

We want to be able to use and import our code for model generation so it needs to be in a module instead of a notebook in which it is now.

For now, this is random search.
Investigate whether the codes works and how long the runs take for one model.

Brainstorm about architure (the engine of the tool), e.g.: did we implement all essential functionlaties?

Y Zhengh proposed in 2014 that splitting multi-variate time series into univariate signals and processing them seperately as distinct branches of the DL architecture is better than processing them as a multi-variate CNN... I am not sure whether this makes sense, but it should be something we can easily test in Keras.

The article is on the onedrive, link: https://nlesc-my.sharepoint.com/personal/v_vanhees_esciencecenter_nl/_layouts/15/guestaccess.aspx?guestaccesstoken=cKHpfUmasCukMxT9YMnoLKvwtQiFlFYdJclcl%2buhcYM%3d&docid=17139ecaca7d5428ea3d184e04a4e59f5

Sometimes learning stops with a completely stable accuracy and loss on val and test set. Find out why the process is stuck.

• Find cases when this is happening and save them so they can be replicated
• Find out why it is stuck in the current state (0 gradient?)
• Find out how the process got to the current state (vanishing or exploding gradient? inappropriate learning rate?)

for now this will be some kind of numpy array. Document somewhere what the exact format is
includes:
data
labels
labels to class name mapping

comment in this item the location where this is documented
also file format

http://karpathy.github.io/2015/05/21/rnn-effectiveness/

In the CNN model, each convolution should be followed by a batch normalization layer before the activation layer. Look up details about batch normalization in for instance the dense layers.

• Model generation function updated
• Write test or think of good reason why not
• Manually test if anything can be trained now
  Bonus points:
• Compare performance of old CNN code with BN version. Did BN version at least improve a bit?

Implement kNN
and add a function in optimal model search where kNN is applied to compjare with the best deep learning model,

Implement DTW and integrate it into the optimal model finding.

What experiments are we going to do to show the value of our tool?

We should not assume that incoming data is normalized. The user should not be responsible for that as it requires a bit of expert knowledge to know normalization is important and how it should be done. An easy way to handle and automate this is to start each model with a batch normalization layer.

Done after:

• Add normalization to the module (as a layer or otherwise)
• Write tests
• Add something to the documentation about this

For example, classify a fixed length numeric time series into classes. No visualisation at this stage, but clear report on loss, accuracy and hyper-parameter optimization.
End product: Document with written description of the deliverable, where possible clarified with diagrams/tables and add to gitbook

SVG or similar that shows all learning curves together based on json output of training process. Models can be filtered with checkboxes for example.

Reflect on progress and review the feature requirement list

I think the current number is way too low.

Wrap up code as a Python module and github documentation as a starting point for generic functionality

Don't return as json or similar the histories of all training processes. Write them to disk during or between training processes.

Let's try to at least get the same result as kNN/RF.

setup:
travis-ci
some code coverage tool
some code analysis tool

insert badges for all 3 into readme

Input:

• List with compiled model objects (predefined architectures and hyperparameters in keras)
• data, start with UCR data

Output:

• Performance per model object (bv accuracy, plots, sklearn, ....)
• Trained weights per model object that can be used for unseen data classification