uber / bayesmark Goto Github PK
View Code? Open in Web Editor NEWBenchmark framework to easily compare Bayesian optimization methods on real machine learning tasks
License: Apache License 2.0
Benchmark framework to easily compare Bayesian optimization methods on real machine learning tasks
License: Apache License 2.0
bayesmark/build_wheel.sh is using sdist command and therefore not building a wheel file
Put the import of gitpython
in a try-except because it is not needed for the version number when installing off the wheel (PyPI
).
Allow the user to use yaml
instead of json
, if they prefer, for the config.json
file.
I'd love to see an integration of HEBO as baseline right into this benchmark. It seems to be the strongest baseline right now. :)
Allow the user to call experiments with an argument file: commands can be called like
bayesmark-exp --args args.yaml
instead of specifying arguments. All the arguments are simply in the yaml
file instead.
Instead of using the sklearn models, is it possible to add new optimization problems in bayesmark?
Issue: Running Bayesmark on Windows using Cygwin throws an exception in cmd_parse.py. This is because PosixPath does not work in Windows.
Resolution: Instead of using PosixPath, use Path instead. Instantiating Path creates either a PosixPath or a WindowsPath depending on the OS. See here: https://docs.python.org/3/library/pathlib.html#pathlib.Path
Thanks for a great project. I'm using the benchmarking function in bayesmark/sklearn_func.py
. This file defines several benchmark functions using sklearn
. A "space" is defined for each function's hyperparameters. How is this "space" determined?
# RF
rf_cfg = {
"max_depth": {"type": "int", "space": "linear", "range": (1, 15)},
"max_features": {"type": "real", "space": "logit", "range": (0.01, 0.99)},
"min_samples_split": {"type": "real", "space": "logit", "range": (0.01, 0.99)},
"min_samples_leaf": {"type": "real", "space": "logit", "range": (0.01, 0.49)},
"min_weight_fraction_leaf": {"type": "real", "space": "logit", "range": (0.01, 0.49)},
"min_impurity_decrease": {"type": "real", "space": "linear", "range": (0.0, 0.5)},
}
In particular, I'd like to know how {log, logit, bilog} is determined here. Is there anything that would help me?
It is not considered best practice to use install_requires to pin dependencies to specific versions, or to specify sub-dependencies (i.e. dependencies of your dependencies). This is overly-restrictive, and prevents the user from gaining the benefit of dependency upgrades.
https://packaging.python.org/discussions/install-requires-vs-requirements/
Currently, the versions of install_requires
and extra_requires
are pinned. But it's inconvenience for users. I propose that specifies what a project minimally needs to run correctly.
Hello, just mentioning that the option used for Nevergrad in your plots is OnePlusOne.
As pointed out during early stages of BayesMark, OnePlusOne is certainly not the best tool for your setting. So I would recommend using NGOpt or, even better, compare several options (CMA, NGOpt, CMandAS2, DE, PSO, GeneticDE, cGA).
Have a great 2022!
I found no explanation of the visible_to_opt
and generalization
in the documentation, are they some kind of linear transformation to the normalized mean score?
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.