ml-opt / dnn_opt Goto Github PK

We need to create a notebook to compare gradient based training of MLP networks with meta-heuristic training. We suggest to proceed as follows:

1. Use the datasets provided by #11 to perform the comparison.
2. Use a fixed sized architecture for the MLP (for example, single hidden layer with 10 neurons using logistic activation function).
3. Consider meta-heuristic algorithms PSO, FA and CS. Hyper-parameter configuration of such algorithms should be archived by using Random Search.
4. Consider the gradient based algorithms: SGD and ADAM provided by skit-learn. Hyper-parameter configuration of such algorithms should be archived by using Random Search.
5. Take a look to hyper-opt to perform hyper-parameter configuration.

The output of this issue should be a new folder within docs directory.

• Name the new directory as: docs/bench-accuracy
• Include within this folder the mlp-sgd-vs-meta.ipynb and mlp-sgd-vs-meta.html notebook.

This issue is a continuation of issue #10.

We need to implement the following test functions in parallel using CUDA, that is in cudalayer:

• Ackley
• Alpine
• De Jung
• Griewangk
• Rastrigin
• Rosenbrock
• Schwefel
• Step
• Styblinski_tang
• Brown Function
• Chung Reynolds Function
• Cosine Mixture Function
• Csendes Function
• Deb 1 Function
• Deb 3 Function
• Dixon & Price Function
• Egg Holder Function
• Exponential Function
• Giunta Function

Each function specification can be found in: http://arxiv.org/abs/1308.4008v1

The scope of this issue is parallel implementation using CUDA in the cuda layer.

Important: remember to modify the solution folder structure according to the modifications done in #6 - #7 (create bench folder and namespace).

We need to create a notebook comparing the efficiency, speed-up and scalability of sequential and OMP implementations of the standard benchmark functions. For this comparison, we should consider the following benchmark functions:

• Ackley
• Alpine
• De Jung
• Griewangk
• Rastrigin
• Rosenbrock
• Schwefel
• Step
• Styblinski_tang
• Brown Function
• Chung Reynolds Function
• Cosine Mixture Function
• Csendes Function
• Deb 1 Function
• Deb 3 Function
• Dixon & Price Function
• Egg Holder Function
• Exponential Function
• Giunta Function

In addition, consider to perform the experiments using different solution vector sizes (from $1.0·10^2$ to $1.0·10^6$) and different hardware configurations (for example, Core i3, Core i7 and CEMC-Cluster).

This output of this issue should be a new folder within docs directory.

• Name the new directory as: docs/bench-time
• Include within this folder the stdf-seq-vs-omp.ipynb and stdf-seq-vs-omp.html notebook.

Provide a sequential implementation for the algorithm Gray Wolf Optimization introduced by:

MIRJALILI, Seyedali; MIRJALILI, Seyed Mohammad; LEWIS, Andrew. Grey wolf optimizer. Advances in engineering software, 2014, vol. 69, p. 46-61.

See here

I want to implement a series of test functions to validate the performance of the optimization algorithms used in the dnn_opt library.

This issue is a continuation of issue #6.

We need to implement the following test functions in parallel using OMP, that is in coptlayer:

• Brown Function
• Chung Reynolds Function
• Cosine Mixture Function
• Csendes Function
• Deb 1 Function
• Deb 3 Function
• Dixon & Price Function
• Egg Holder Function
• Exponential Function
• Giunta Function

Each function specification can be found in: http://arxiv.org/abs/1308.4008v1

The scope of this issue is parallel implementation using OMP in the copt layer.

Important: remember to modify the solution folder structure according to the modifications done in #6 - #7 (create bench folder and namespace).

• Change the name of the file_reader class to csv as this will be more intuitive for users
• Include features to parse files that are delimited by a specified separator
• Remove current file header that consist in number of instances, input dimension and output dimension. The first one can be captured directly from file lines given this specific format (csv) and the former two can be captured from constructor

We need to include a set of benchmark datasets to compare the accuracy and efficiency of our neural implementations. In this case, we are looking for small-size datasets (less that 1k examples) that deals either classification or regression. Probably, a good starting point are the following publicly available datasets:

• Breast cancer
• Contact lenses
• CPU
• Credit
• Diabetes
• Glass
• Ionosphere
• Iris
• Labor
• Soy bean
• Supermarket
• Unbalanced
• Vote
• Wheather

We need to create a notebook transforming the data as follows:

• Dataset format should be CSV with header information (coma-separated, header contains attribute names).
• Nominal and ordinal values should be transformed into numerical one (create dummies for simplicity).
• Normalize numerical values (scale the values to [0, 1])
• Ordinal cyclic variables such as day of the week should be transformed using sin and cos transformations.
• Remove missing data.
• Identify and remove outliers.

Every steep should be documented and explained in the notebook. Include in the notebook the following information for each dataset:

• Number of input and output variables.
• Variable type: nominal, ordinal, numeric, etc.
• Number of examples.
• Are the examples balanced or not?
• Number of examples removed due to missing values.
• Number of outliers.
• Best reported accuracy so far, describe the model used (include an ISO 690 reference to the paper).

The output of this issue should be a new directory structure within docs folder. Create the following hierarchy of folders:

• docs/bench_data/small/original/: with all the .ARFF original files.
• docs/bench_data/small/transformed/: with all the transformed files.
• docs/bench_data/small/notebooks: with the transform.ipynb and transform.html notebook that can be used to generate the docs/bench_data/small/transformed/ files from the files in docs/bench_data/small/original/.

Provide a member function to set the seed of any generator

Gradient based algorithms are the default training algorithms for ANN. Hence, providing support for such algorithms (SGD, ADAM, RMSProp, etc.) is critical in order to provide out of the box benchmarking capabilities. Our suggestion is to proceed as follows:

• Create a new class in base package (e.g. class derivable : public solution {} that will inherit from solution class). derivable should have an array of floats (e.g. derivable::m_df) that represents the derivative of the solution::fitness() function respect each parameter in solution::get_params(). Hence, derivable::get_df() size will be solution::size().

• Define a getter in derivable class (e.g. derivable:df()) that, in case the solution was modified will calculate the derivative of the fitness function and store the result in derivable::m_df array. In case the solution is not modified, it will simply return derivable::m_df. The implementation of this method should be the same as in the current solution::fitness() method.

• As in the case of solution::fitness() and solution::calculate_fitness() consider an implementation of derivable:df() and a protected virtual derivable:calculate_df() = 0 method in derivable. It is probably a good idea to do not create derivable::m_df before the first derivable:df() call just in case the derivative gets never used.

• Each child of derivable in solutions package should re-implement its own version of virtual derivable:calculate_df() = 0 according to the fitness function (only if the fitness function is derivable of course). This means that the network class should inherit from derivableinstead of solution and implement virtual derivable::calculate_df() = 0.

• The network:calculate_df() implementation will call a layer::backprop() method defined in the layer class, passing the position in the derivable::m_df array where the layer will store the derivative of its corresponding parameters. layer::backprop() method should be similar to the current layer::prop() method.

• Each child of layer in package layers should re-implement its own version of virtual layer:backprop() = 0. Currently there should be a single layer fc (fully connected layers) implemented in the library.

• Create a new class in algorithms package class sgd : public algorithm that using the derivative and the fitness function of a derivable solution can implement Stochastic Gradient Descent Algorithm.

Include a reader classes to support input files in several standard formats. For the moment, it would be useful to consider:

• CSV format:
• ARFF format:
• LMBD format:
• HDF5 format:
• JSON format:

Please update the list as you implement the reader for each format. In addition, include web links to a description of each format as you proceed. I would like to see a separated comment clarifying how each format was implemented if needed.