Comments (7)
I also found this blogpost very useful:
http://blog.turi.com/how-to-evaluate-machine-learning-models-part-4-hyperparameter-tuning
from mcfly.
And we can look into Optunity they also support for example TPE and other optimizers.
from mcfly.
Ah sorry but I see that Optunity uses hyperopt under the hood for TPE so we might run into the same problems as hyperas (#35)
from mcfly.
Optunity allows for CMA-ES optimizer. According to 'Algorithms for Hyper-parameter Optimizations' by James Bergstra, " CMA-ES is a state-of-the-art gradient-free evolutionary algorithm for optimization on continuous domains, which has been shown to outperform the Gaussian search EDA. Notice that such a gradient-free approach allows non-differentiable kernels for the GP regression."
I struggle to digest this. Does this mean that it can handle non-real numbers as hyperparameter, like we want or is a non-differentiable kernel something different?
from mcfly.
Rescale is a commercial tool to train deep networks in the cloud, including Keras, Torch,... Part of the service is Keras hyperparameter optimization. https://blog.rescale.com/deep-neural-network-hyper-parameter-optimization/ It may be good to know that these services exist.
from mcfly.
In that blogpost, they use SMAC - which trains random forests on the results, and is better on categorical variables according to the blog of Alice Zheng.
SMAC is available in python in the pysmac package
from mcfly.
Another interesting blogpost: http://www.argmin.net/2016/06/20/hypertuning/ (also the comments below)
Conclusion is that bayesian methods such as TPE and SMAC are only somewhat faster in finding an optimum than random search, the speedup is not more than 2x - and random search is easily parallelizable.
It seems that TPE and SMAC are the only algorithms that are really suitable for the type of problem that we have: with mixed categorical, discrete and continuous hyperparameters.
This paper compares the methods. SMAC seems to be better than TPE in a majority of the medium/high-dimensional cases.
from mcfly.
Related Issues (20)
- add cron builds
- fix test_find_best_architecture_with_class_weights
- Update build workflow
- Prepare some regression dataset
- Write a tutorial on regression functionality
- Create a plan for regression implementation HOT 1
- Not all training data in memory at the same time
- advertise that we can do regression in all our docs
- create new release (christmas edition) HOT 1
- explain in the docs how we automatically detect regression/classification data
- implement regression functionality in code
- Adapt defaults for evaluation metrics HOT 1
- find_best_architecture fails with tf.keras.metrics objects
- create sphinx build action for documentation build testing
- update supported python versions in tutorial docs and CI workflow HOT 2
- throw sensible error when using a datagenerator and subset!=None
- Add Transformer architecture
- Early stopping a bit aggressive for low number of epochs
- We should shuffle the data by default
- Consider this architecture TCNForecaster (2018)
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from mcfly.