• Using hierarchical subsampling?
• "Manual" way of doing it: Get time of day distributions from large but low sample rate datasets, use that to generate infinite amounts of training data by combining tracebase waveforms

Could try to build single system which can cope with many different sample rates from kHz to hourly data. Top few layers are common (and hence, if they are recurrent, might need to output at a sample rate which is the lowest common sample rate across all datasets, say 2 minutes) . But replace bottom layers for different sample rates. I suppose this is a form of ‘transfer learning’ except I want bi-directional transfer. Maybe we need to concurrently train multiple parallel lower layers (each for a different ‘type’ of input, e.g. kHz, 1 second, 10 second, 1 minute, 2 minute, 15 minute, with or without reactive power etc) all connected to a single upper layer. Training would be interleaved. This would perhaps force the upper layer to learn the common properties. NIPS 2014 paper on multimodal deep learning

See if this means we can get performance like e59a, but without one or both of the first dense layers.

Also try using linspace to init weights of first layer. And try visualisation to see what those neurons are doing (#15)

• Compute NILM metrics on every validation cycle
• Create benchmarks using FHMM and CO and Hart on, say, 1 month of data. Use this to compare against NeuralNILM

• modify Source superclass so it can take a input_preprocessing=[] argument
• Each Preprocessing class has
  • input : the data source
  • get_output_shape() for method to allow Source to figure out the final output shape
  • get_output()
• pass input data through this pipeline, and modify n_inputs and seq_length.

Train first on synthetic data where we just have, say, 500 timesteps, half of which are the TV. Mix with a combination of a small number of other appliances and nothing. Ignore time-of-day. Then, once these weights are trained, maybe add in time-of-day, possibly feeding into an upper layer.

Ideas:

• Find a minimal example which still fails (something that trains fast)
• Is it:
  • The new nntools code (check by re-running e82)
  • dodgy data (check using old_X and old_y)
  • initialisation?
  • learning rate?
  • different learning algorithm?
    • Try adagrad?
    • Try skaae's adadelta_normscaled
    • "rescale the learning rate according to the difference in cost normalization"
  • try concat layer (see this comment)
  • Different cost function.
    • Read this about categorical_crossentropy. Perhaps I should be using the binary_crossentropy (their simple auto-associator sounds vaguely similar to what I'm doing). - oops, we are already using the binary_crossentropy!
    • Try MSE
  • doesn't like producing 5 outputs?
    try single-appliance experts.
  • smaller seq length?
  • smaller batch size?
  • doesn't like range of targets (maybe shouldn't get right to 1?)
  • quantized inputs
  • do we get NaNs if we remove the Conv1D layers?
  • do we get NaNs if we swap from BLSTM to LSTM?
  • do we get NaNs if we remove (B)LSTM layers?
• Hacks:
  • Save, say, last 5 training examples. When we hit a bad patch, examine these examples.
  • Save network weights. If we get a bad bit of training then revert network to, say, 5 steps ago.

• Dr Alexander Gluhak (from Intel ICRI) is interested in multi-model disaggregation (e.g. using water and energy). Maybe it would be fairly simple to extend a neural net to use multi-model data?
• Use weather data from my house
• AMPds has water data, I think. Some other datasets might, too.

Once I have finished building an awesome disaggregation algo, release it as a really simple web service so others can easily benchmark against it.

also:

• have both inputs in range [-1, 1](try different ranges)
• Centre mains power

• weight noise
• input noise
• dropout

probably not possible?

• protects again graphic machine being rebooted
• related: #15

Output power demand and also information about the appliance. Could build a hierarchical appliance classification system. Related: #13

Use two networks: one outputs boolean for if the appliance is on or off. The other takes two inputs: the aggregate power demand; and whether the appliance is on or off; and outputs the disag power demand. Try doing this as two separate networks (trained separately) and also try as a single network (where we have a hidden layer which gives the boolean on/off signal)

by getting it to predict the next value

• Supervised pre-training
• Unsupervised using auto encoder

Related:

• #3

Representation:

• quantized
  • one-hot
  • many-hot
  • signed (if using fdiff)

Inputs:

• fdiff
• two inputs: t and t+1 (so network can do diff on its own)
• time of day
• day of week / business day
• time of year
• country
• weather (either from dataset or from met office)

• use more appliances from house 1
• appliances from other houses
• train on one set of houses, test on another

• compare shallow net using my hand engineered feature detectors as inputs vs deep net without hand engineered detectors. Could try getting a net to just reproduce each of my feature detectors.
• Compare RNN vs convnet.

• a static visualisation which shows the ‘receptive field’ of each neuron
• dynamic visualisation which shows which neurons are on at different time steps in processing an input. Use a matplotlib slider to allow users to scrub back/forth. See this Lasagne issue for some hints about visualisation.

• try sigmoid activation for Conv1D layer
• Try ReLU (it appears that I can't get ReLU to work in any place in the network! I've tried in the Conv1DLayer and the DenseLayers)
• convnet (stride 1), then LSTM
• Try using input in range from minus 1 to plus 1
• convnet (stride 1), convnet (stride 1), then LSTM
• Try convnet (stride 1), then fully connected, then LSTM.
• layerwise pretraining for the ConvNet layers (autoencoder) #4
• Try larger range
• Read about initialisation for rectifier units
• Try with no LSTM layer and boolean targets.
• Convnets should be easy to visualise
• Try a purely feed-forward network using 1D convolutions and pooling. Zhang & LeCunn Text understanding from Scratch (2015) is a fascinating example of this working well. Also see Lecun 1998.
• Use linear space to init weights and bias for bottom layer?
• #17

#21

Wait until appliance has finished its run, then output the total energy usage for that appliance (in one go). Or wait until end of sequence (but that requires long memory).

Also need to modify NILMTK so we can do training and disag using simple dataframes.

Shrink down to a hidden layer with K units, where K is the number of appliances.
But how to do RNN auto encoder? Generate sequence after a delay? Generate k-length window given a k-length window of data?

Related:

• #3

As per the Deep Mind tutorial on 18th Feb. Produces probabilistic output. Handles missing data very well. Both things we need!

http://arxiv.org/abs/1401.4082

Try getting the net to parameterise a mixture of Gaussians for probabilistic output. Maybe also output appliance estimates as proportion of total energy

Or maybe output a CSV file with train cost, validation cost, NILM metrics, secs per epoch, train weights etc. Start log with config of the network, then each training loop

Each directory would be like this:

• e92
  • e92.py (define experiment)
  • e92.h5 (costs, metrics, network weights etc)
  • e92_costs.png (multiple subplots: cross entropy, MSE, NILM metrics)
  • e92_estimates_1250epochs_3.png
• Read .py scripts (one for each experiment) from a directory
• Run each script in sequence.
  • Catch exceptions and log them
• Need a better way to set max_appliance_power, on_duration, off_duration etc. Maybe these should go into the NILMTK metadata. Although, to start with, maybe just use a function to set the metadata manually from the experiment script. I'm also starting to think that we should use real aggregate data now. Our synthetic data doesn't include, for example, that the fridge turns on multiple times.
• Output results from each experiment to an HDF5 file:
  • training costs
  • validation costs (need a standardised validation timeseries, make sure it has a section which is 'easy' i.e. just the appliances on their own)
  • NILM validation costs
  • network weights. Both for analysis / visualisation later and also to allow training to be restarted (if power is lost) and also to use the net.

Use small sequences (100). See page 9 in Graves 2014.

This looks like a good tutorial on backprop through time (BPTT) which I think is what Graves uses? The tutorial has a brief mention of truncating the gradient, although I think Graves does it in a different way.

Janczak. Identification of Nonlinear Systems Using Neural Networks and Polynomial Models. 2005 might also have some useful info

Does skaee mention this in his code or papers?

Does Graves do it on "micro batches" of 100 or does he do a sliding window? Reread Graves 2014. Check his book. Check the PhD thesis he cites. Check his code.

Perhaps it's as simple as splitting data into batches and have an option to allow activations to persist between sequences or between batches.

http://arxiv.org/abs/1502.03167

• maybe just set the network up so it gets a sample every second. Missing samples we run the network but with no input (i.e. just recurrent inputs)
• or we give a ‘time index’ input. This might be handled as a special input, somehow directly influencing the LSTM blocks???

http://en.wikipedia.org/wiki/Ensemble_learning

Just as soft max allows outputs to form a categorical distribution, maybe there is some way to force the outputs to sum to no more than the aggregate power demand at each time slice. If the network outputs proportion of energy per appliance then perhaps we can just use soft max. But what about unrecognised appliances? And can we still use density mixture models? And maybe getting it to estimate proportion won't work well if we use high frequency voltage data. Also means that the output for every appliance would have to change when any appliance changes state, which seems like a bad idea

Vary one parameter (keeping others constant), pick best, vary next parameter, pick best, go round in loop

A Clockwork RNN. Koutnik, Greff, Gomez & Schmidhuber 2014. Could be good for multiple datasets at different temporal resolution (#7)

Graves does this. See Graves 2014. I think skaae has implemented this in his code.

• energy usage for entire activation
• rolling average
• fdiff
• raw power demand
• quantized raw power demand / fdiff
• classification:
  • on / off binary
  • multiple discrete states per appliance. Perhaps these could be found by a separate net whose aim is to reconstruct the ground truth from the ground truth

Related: #19