joeddav / devol Goto Github PK

Looks like an awesome project!

We've found a fun use case for deep learning for regression problems with auto_ml. We train the neural network, then instead of getting it's output from the final layer (just a linear model), we get the features it learned in it's penultimate layer, then feed those into a gradient boosted model (which is typically better at turning features into predictions than a linear model).

Your library looks like a great way to optimize the deep learning model.

Any thoughts on what it would take to get regression models supported?

Future models can't even compile when the model before it runs out of memory, resulting in termination of the entire process. Something to do with GPU memory allocation.

In the mean time, try not to allow more parameters than your system is equipped to train.

Hi,
I have been trying to use devol but to no avail. It seem that everything has been installed correctly. However, when I try to check the example, I constantly receive an error.

Python 3.5.6 |Anaconda, Inc.| (default, Aug 26 2018, 21:41:56)
[GCC 7.3.0] on linux
Type "help", "copyright", "credits" or "license" for more information.

from devol import DEvol, GenomeHandler
Traceback (most recent call last):
File "", line 1, in
File "/home/luferrolino/devol/devol/init.py", line 1, in
from devol import DEvol
ImportError: cannot import name 'DEvol'

pip show devol gives me this

(devol4) luferrolino@luferrolino-Predator-G3-571:~$ pip show devol
Name: devol
Version: 0.2
Summary: Genetic search for CNN classifier in Keras
Home-page: https//github.com/joedav/devol
Author: Joe Davison
Author-email: [email protected]
License: MIT
Location: /home/luferrolino/devol
Requires: keras
Required-by:

line 76 77
86 87
and 132 133 have inconssistant usage of space and tab for indent
on the genome_handler.py

I would like to test the model with the Sample datasets.

with open(self.datafile, 'a') as csvfile:
OSError: [Errno 22] Invalid argument: 'Thu Apr 11 00:42:22 2019.csv'

This file can not be made for me. Any solutions for that?

I am trying to use this code for some different data-set, in which image sizes are different (75 x 75) and two channels are there. But It stops working in the beginning with a pop window which states that "python stopped working".

Here are some other details (I modified the demo code).
num_classes = 2

x_train.shape = (1203, 75, 75, 2)
y_train = (1203, 2)
x_test = (401, 75, 75, 2)
y_test = (401, 2)

Rest of the parameters are all same. Image is in 'float32' with normalized between 0 - 1.

Any help??

I have a simple toy rnn lstm in keras. To run it through devol I set the convolution variables (max_conv_layers, max_filters) to 0 because I am not doing any convolutions. This obviously (in hindsight) triggered a division by zero error (or similar). Point being, running this just for a sequential dense model seems to break it. Scanning through the genome_handler.py file I see that convolution layers are required - they're absence is certainly not tested for.

This is an awesome help. I would like to be able to fine tune an already existing model using this algorithm.

Would it be possible that I feed in an existing model and start creating next generation models from it?

Also is it possible to add LSTM into the mix?

Hey,

I have tried the github way of installing repositories:
python setup.py install
it says installed but i cannot import :
import devol
It says:
ImportError: No module named 'devol'
when trying to execute the demo notebook

What could be going wrong?

Hey the algorithm is awesome, but I was wondering if there is a way to continue from the csv, so using it as a checkpoint of the evolution. Do you think that is possible, or is it already implemented and I missed it?

Thank you very much!

Hello,
I was trying to implement the "hello world" example and exploring the GenomeHandler function parameters.
I was able to put the example running but once I set the max_pooling=False I get the error 'a' must be greater than 0 unless no samples are taken on the devol.run function

Can someone help me on that?

Code

(x_train, y_train), (x_test, y_test) = mnist.load_data()

x_train = x_train.reshape(x_train.shape[0],x_train.shape[1], x_train.shape[2], 1).astype('float32') / 255
x_test = x_test.reshape(x_test.shape[0], x_test.shape[1], x_test.shape[2], 1).astype('float32') / 255
y_train = to_categorical(y_train)
y_test = to_categorical(y_test)
dataset = ((x_train, y_train), (x_test, y_test))

genome_handler = GenomeHandler(max_conv_layers=6 ,max_dense_layers=2,max_filters=256,max_dense_nodes=1024,input_shape=x_train.shape[1:],n_classes=10, batch_normalization=True, dropout=True, max_pooling=False)
devol = DEvol(genome_handler)
model = devol.run(dataset=dataset, num_generations=1,pop_size=1,epochs=1)

Error

ValueError                                Traceback (most recent call last)
<ipython-input-36-9cc5827c046e> in <module>
      2                   num_generations=1,
      3                   pop_size=1,
----> 4                   epochs=1)

c:\users\joao\documents\devol\devollib\devol\devol.py in run(self, dataset, num_generations, pop_size, epochs, fitness, metric)
    120 
    121         # generate and evaluate initial population
--> 122         members = self._generate_random_population(pop_size)
    123         pop = self._evaluate_population(members,
    124                                         epochs,

c:\users\joao\documents\devol\devollib\devol\devol.py in _generate_random_population(self, size)
    233 
    234     def _generate_random_population(self, size):
--> 235         return [self.genome_handler.generate() for _ in range(size)]
    236 
    237     def _print_result(self, fitness, generation):

c:\users\joao\documents\devol\devollib\devol\devol.py in <listcomp>(.0)
    233 
    234     def _generate_random_population(self, size):
--> 235         return [self.genome_handler.generate() for _ in range(size)]
    236 
    237     def _print_result(self, fitness, generation):

c:\users\joao\documents\devol\devollib\devol\genome_handler.py in generate(self)
    208             for key in self.convolutional_layer_shape:
    209                 param = self.layer_params[key]
--> 210                 genome.append(np.random.choice(param))
    211         for i in range(self.dense_layers):
    212             for key in self.dense_layer_shape:

mtrand.pyx in mtrand.RandomState.choice()

ValueError: 'a' must be greater than 0 unless no samples are taken

In genome_handler.py

dim = min(self.input_shape[:-1]) # keep track of smallest dimension

The [:-1] causes an error when the input is flat (i.e. not an image), as it tries to take a min over a empty variable.

For me changing to

dim = min(self.input_shape]) # keep track of smallest dimension

fixed the issue.

An the end of all the runs of the demo, there's a traceback...

Train on 60000 samples, validate on 10000 samples
Epoch 1/1
60000/60000 [==============================] - 54s - loss: 0.2622 - acc: 0.9143 - val_loss: 2.6748 - val_acc: 0.3584
Generation 10: best accuracy: 0.9853 average: 0.8218 std: 0.3026
Traceback (most recent call last):
File "test.py", line 48, in
model, loss, accuracy = devol.run(dataset, num_generations, population_size, num_epochs)
File "/home//projects/devol/devol.py", line 107, in run
return self.genome_handler.decode_best(self.datafile)
File "/home//projects/devol/genome_handler.py", line 217, in decode_best
return self.decode(self.best_genome(csv_path, metric, False))
File "/home//projects/devol/genome_handler.py", line 209, in best_genome
row = list(data[:, col]).index(best(data[:, col]))
ValueError: nan is not in list

First of all, nice work @joeddav . I would like to know if you could explain the mutate function over genome_handler.py. Because I would like to set mutate rate like in population from crossover (95%) and best models survive rates (95%).

Hello,

I would like to share an idea I recently had.

Currently, I have the feeling that there are two kinds of auto-ml tools: those which build neural networks (and yours looks very promising) and those which tune and combine classifiers or regressors (auto-sklearn and TPOT for instance).

I think we would benefit from using both in some cases. For example, suppose I want to classify images. I can't use auto-sklearn or TPOT directly on images. However, I could:

1. Use DEvol to train a multilayer autoencoder which minimizes the reconstruction error
2. Use this trained autoencoder as a sklearn transformer to retrieve the extracted features (in the middle layer)
3. Provide those features to TPOT and/or auto-sklearn

The final result is a sklearn pipeline which has been built automatically from the raw dataset. In a sense, it is similar to the idea @ClimbsRocks discussed in #15. But instead of using only a gradient boosted model with the extracted features, we just let auto-sklearn and/or TPOT find the best combination of models.

What do you think about this idea? How much work would be necessary to allow DEvol to build autoencoder?

Hi,

as mentioned in the ReadME. This can be applied to other problems. I would like to apply that on image segmentation which has upsampling layer as well as conv, dropout, relu activation function, but I think this is not possible to do so. I couldn't see a way to do that.

Best,

Hey,
I tried to run a code similar to Your demo.ipynb. After running this:

num_generations = 2
population_size = 2
num_epochs = 1

devol = DEvol(genome_handler)
model = devol.run(dataset, num_generations, population_size, num_epochs)
model.summary()

Error occurred:

Genome encoding and accuracy data stored at Thu Jan 18 02:28:02 2018.csv 

---------------------------------------------------------------------------
OSError                                   Traceback (most recent call last)
<ipython-input-7-68908626ba10> in <module>()
      3 num_epochs = 1
      4 
----> 5 devol = DEvol(genome_handler)
      6 model = devol.run(dataset, num_generations, population_size, num_epochs)
      7 model.summary()

e:\python\src\devol\devol.py in __init__(self, genome_handler, data_path)
     33 
     34         print("Genome encoding and accuracy data stored at", self.datafile, "\n")
---> 35         with open(self.datafile, 'a') as csvfile:
     36             writer = csv.writer(csvfile, delimiter=',', quotechar='"', quoting=csv.QUOTE_MINIMAL)
     37             genome = genome_handler.genome_representation() + ["Val Loss", "Val Accuracy"]

OSError: [Errno 22] Invalid argument: 'Thu Jan 18 02:28:02 2018.csv'

I'm using Python 3.6
Windows 10.

When running the demo.ipynb after updating only these variables...

num_generations = 20
population_size = 3

I got the following error after Generation 13.

---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
mtrand.pyx in mtrand.RandomState.choice (numpy/random/mtrand/mtrand.c:16934)()

TypeError: 'float' object cannot be interpreted as an integer

During handling of the above exception, another exception occurred:

ValueError                                Traceback (most recent call last)
<ipython-input-45-a3c0692feb14> in <module>()
     42 
     43 devol = DEvol(genome_handler)
---> 44 devol.run(dataset, num_generations, population_size, num_epochs)

/home/ubuntu/devol/devol.py in run(self, dataset, num_generations, pop_size, epochs, fitness)
     41             members += pop.getBest(pop_size - int(pop_size*0.95))
     42             for i in range(len(members)): # Mutation
---> 43                 members[i] = self.mutate(members[i], gen)
     44             fit = []
     45             for i in trange(len(members), desc="Gen %i Models Fitness Eval" % (gen + 1)):

/home/ubuntu/devol/devol.py in mutate(self, genome, generation)
     73     def mutate(self, genome, generation):
     74         num_mutations = max(3, generation / 4) # increase mutations as program continues
---> 75         return self.genome_handler.mutate(genome, num_mutations)
     76 
     77 class Population:

/home/ubuntu/devol/genome_handler.py in mutate(self, genome, num_mutations)
     71 
     72     def mutate(self, genome, num_mutations):
---> 73         num_mutations = np.random.choice(num_mutations)
     74         for i in range(num_mutations):
     75             index = np.random.choice(range(1, len(genome)))

mtrand.pyx in mtrand.RandomState.choice (numpy/random/mtrand/mtrand.c:17018)()

ValueError: a must be 1-dimensional or an integer

Hi
This Library is a great idea and has a lot of potential for model selection! Thank you!

I have been adding a couple of enhancements to my fork. Essentially I just wanted to use validation loss as a metric and ended up adding lots of other stuff as well. I'm still thinking my revisions are messy and obviously contain some breaking changes but I'm happy to test and submit individual pull requests per change if any of the changes below are helpful:

1. GenomeHandler is passed a metric default is "accuracy" which sets maximizing or minimizing objective for DEvol so we can optimize validation loss instead of accuracy if we choose. ( In actual fact this whole thing could be passed directly on init to DEvol if it was not referenced in the GenomeHandler in best_genome, but that would probably not make sense).

2. I added a flatten_layers param to GenomeHandler so that we include the global max and average pooling in the search. the parameter name flatten_layers is horrible through, I am trying to come up with something better.

3. Added an arg for DEvol to fit verbose, It was something I wanted and I noticed it was in the projects section. It just calls fit with verbose=1 and doesn't look so hot with the tqdm so maybe not what you're looking for.

4. I added nadam, vanilla sgd and sgd with momenum, nesterov, & decay to optimizers. I'm not sure that this is really a nessesity since users could add themselves , but it's nice to have sgd especially to compare to newer (possibly less stable) optimizers.

5. added Elu to list of activations. As above could be argued that it doesn't need to be a default (& mixed results with batchnorm so...)

I also just wanted to ask about the score method of Population class:

    def score(self, fitness):
        return (fitness * 100)**4

I was just a little confused as to why the values needed to be so large, where we are already min/max scaling fitnesses? I have left this out (using only scaling) and am not seeing a difference. I thought I must be missing something so I wanted to ask.

Can please modify the print statement to have it compatible with python 3.X format print() in devol.py

Hi,

What are the column names on the csv file. I think it should be written before the numeric values.

Thanks !

Genome encoding and accuracy data stored at Wed Jan 24 15:38:15 2018.csv

model 1/20 - generation 1/10:

An error occurred and the model could not train. Assigned poor score.

RuntimeError Traceback (most recent call last)
in ()
6
7 devol = DEvol(genome_handler)
----> 8 model = devol.run(dataset, num_generations, population_size, num_epochs)
9 model.summary()
10

/home/thomas/dsa-experimental/thomas/imagification/Ensemble_Testing/devol/devol.py in run(self, dataset, num_generations, pop_size, epochs, fitness, metric)
79 print("\nmodel {0}/{1} - generation {2}/{3}:\n"
80 .format(i + 1, len(members), 1, num_generations))
---> 81 res = self.evaluate(members[i], epochs)
82 v = res[metric_index]
83 del res

/home/thomas/dsa-experimental/thomas/imagification/Ensemble_Testing/devol/devol.py in evaluate(self, genome, epochs)
143 pass
144 self.bssf = met
--> 145 model.save('best-model.h5')
146
147 return model, loss, accuracy

/usr/local/lib/python2.7/dist-packages/keras/engine/topology.pyc in save(self, filepath, overwrite, include_optimizer)
2554 """
2555 from ..models import save_model
-> 2556 save_model(self, filepath, overwrite, include_optimizer)
2557
2558 def save_weights(self, filepath, overwrite=True):

/usr/local/lib/python2.7/dist-packages/keras/models.pyc in save_model(model, filepath, overwrite, include_optimizer)
114 else:
115 model_layers = model.layers
--> 116 topology.save_weights_to_hdf5_group(model_weights_group, model_layers)
117
118 if include_optimizer and hasattr(model, 'optimizer'):

/usr/local/lib/python2.7/dist-packages/keras/engine/topology.pyc in save_weights_to_hdf5_group(f, layers)
2850 g = f.create_group(layer.name)
2851 symbolic_weights = layer.weights
-> 2852 weight_values = K.batch_get_value(symbolic_weights)
2853 weight_names = []
2854 for i, (w, val) in enumerate(zip(symbolic_weights, weight_values)):

/usr/local/lib/python2.7/dist-packages/keras/backend/tensorflow_backend.pyc in batch_get_value(ops)
2200 """
2201 if ops:
-> 2202 return get_session().run(ops)
2203 else:
2204 return []

/usr/local/lib/python2.7/dist-packages/tensorflow/python/client/session.pyc in run(self, fetches, feed_dict, options, run_metadata)
887 try:
888 result = self._run(None, fetches, feed_dict, options_ptr,
--> 889 run_metadata_ptr)
890 if run_metadata:
891 proto_data = tf_session.TF_GetBuffer(run_metadata_ptr)

/usr/local/lib/python2.7/dist-packages/tensorflow/python/client/session.pyc in _run(self, handle, fetches, feed_dict, options, run_metadata)
1047 raise RuntimeError('Attempted to use a closed Session.')
1048 if self.graph.version == 0:
-> 1049 raise RuntimeError('The Session graph is empty. Add operations to the '
1050 'graph before calling run().')
1051

RuntimeError: The Session graph is empty. Add operations to the graph before calling run().

I get the error No module named 'tqdm' when I install this repo using pip. Can you have it listed as a dependency ?

Hi,
I am really interested in your code and have adapted it for my own purposes. However I have just noticed that the convolutional window is not varied.... This is a particularly crucial factor for my own data.....
Could you recommend any immediate quick fix that could allow for different convolutional windows to be tried? and for such traits selected for/against and hence to be passed onto the next generation etc....
Is this at all possible?
Thanks

This is a great. For this to be immediately useful the devol.run() function should return or save the model that had the lowest validation error. This model can then be trained to completion or reinitialized and retrained.

I figured out how to recreate the model but it still requires some tinkering to get it to model.predict(x_test) since the soft max layer is not included.

f = open('Sun May 28 09:53:45 2017.csv')
genomes = f.read().split('\n')
line = genomes[-2] #get last saved model
last = line.split(',')
last = [int(i) for i in last[:-2]] #only get the model parameters.
model = genome_handler.decode(last)
model.summary()

Genome encoding and accuracy data stored at Wed Jan 24 16:22:07 2018.csv

model 1/20 - generation 1/10:

ValueError Traceback (most recent call last)
in ()
4
5 devol = DEvol(genome_handler)
----> 6 model = devol.run(dataset, num_generations, population_size, num_epochs)
7 model.summary()

/home/thomas/dsa-experimental/thomas/imagification/Ensemble_Testing/devol/devol.pyc in run(self, dataset, num_generations, pop_size, epochs, fitness, metric)
79 print("\nmodel {0}/{1} - generation {2}/{3}:\n"
80 .format(i + 1, len(members), 1, num_generations))
---> 81 res = self.evaluate(members[i], epochs)
82 v = res[metric_index]
83 del res

/home/thomas/dsa-experimental/thomas/imagification/Ensemble_Testing/devol/devol.pyc in evaluate(self, genome, epochs)
114
115 def evaluate(self, genome, epochs):
--> 116 model = self.genome_handler.decode(genome)
117 loss, accuracy = None, None
118 try:

/home/thomas/dsa-experimental/thomas/imagification/Ensemble_Testing/devol/genome_handler.pyc in decode(self, genome)
121 genome[offset + 1], (3, 3),
122 padding='same')
--> 123 model.add(convolution)
124 if genome[offset + 2]:
125 model.add(BatchNormalization())

/usr/local/lib/python2.7/dist-packages/keras/models.pyc in add(self, layer)
462 # and create the node connecting the current layer
463 # to the input layer we just created.
--> 464 layer(x)
465
466 if len(layer.inbound_nodes[-1].output_tensors) != 1:

/usr/local/lib/python2.7/dist-packages/keras/engine/topology.pyc in call(self, inputs, **kwargs)
574 'layer.build(batch_input_shape)')
575 if len(input_shapes) == 1:
--> 576 self.build(input_shapes[0])
577 else:
578 self.build(input_shapes)

/usr/local/lib/python2.7/dist-packages/keras/layers/convolutional.pyc in build(self, input_shape)
132 name='kernel',
133 regularizer=self.kernel_regularizer,
--> 134 constraint=self.kernel_constraint)
135 if self.use_bias:
136 self.bias = self.add_weight(shape=(self.filters,),

/usr/local/lib/python2.7/dist-packages/keras/legacy/interfaces.pyc in wrapper(*args, **kwargs)
85 warnings.warn('Update your ' + object_name + 86 ' call to the Keras 2 API: ' + signature, stacklevel=2)
---> 87 return func(*args, **kwargs)
88 wrapper._original_function = func
89 return wrapper

/usr/local/lib/python2.7/dist-packages/keras/engine/topology.pyc in add_weight(self, name, shape, dtype, initializer, regularizer, trainable, constraint)
398 dtype=dtype,
399 name=name,
--> 400 constraint=constraint)
401 if regularizer is not None:
402 self.add_loss(regularizer(weight))

/usr/local/lib/python2.7/dist-packages/keras/backend/tensorflow_backend.pyc in variable(value, dtype, name, constraint)
383 v._uses_learning_phase = False
384 return v
--> 385 v = tf.Variable(value, dtype=tf.as_dtype(dtype), name=name)
386 if isinstance(value, np.ndarray):
387 v._keras_shape = value.shape

/usr/local/lib/python2.7/dist-packages/tensorflow/python/ops/variables.pyc in init(self, initial_value, trainable, collections, validate_shape, caching_device, name, variable_def, dtype, expected_shape, import_scope, constraint)
211 dtype=dtype,
212 expected_shape=expected_shape,
--> 213 constraint=constraint)
214
215 def repr(self):

/usr/local/lib/python2.7/dist-packages/tensorflow/python/ops/variables.pyc in _init_from_args(self, initial_value, trainable, collections, validate_shape, caching_device, name, dtype, expected_shape, constraint)
320 "construct, such as a loop or conditional. When creating a "
321 "variable inside a loop or conditional, use a lambda as the "
--> 322 "initializer." % name)
323 # pylint: enable=protected-access
324 shape = (self._initial_value.get_shape()

ValueError: Initializer for variable conv2d_4/kernel/ is from inside a control-flow construct, such as a loop or conditional. When creating a variable inside a loop or conditional, use a lambda as the initializer.