uber / bayesmark Goto Github PK

Instead of using the sklearn models, is it possible to add new optimization problems in bayesmark?

Issue: Running Bayesmark on Windows using Cygwin throws an exception in cmd_parse.py. This is because PosixPath does not work in Windows.

Resolution: Instead of using PosixPath, use Path instead. Instantiating Path creates either a PosixPath or a WindowsPath depending on the OS. See here: https://docs.python.org/3/library/pathlib.html#pathlib.Path

bayesmark/build_wheel.sh is using sdist command and therefore not building a wheel file

Add example image results to README

Put the import of gitpython in a try-except because it is not needed for the version number when installing off the wheel (PyPI).

I found no explanation of the visible_to_opt and generalization in the documentation, are they some kind of linear transformation to the normalized mean score?

Thanks for a great project. I'm using the benchmarking function in bayesmark/sklearn_func.py. This file defines several benchmark functions using sklearn. A "space" is defined for each function's hyperparameters. How is this "space" determined?

# RF
rf_cfg = {
    "max_depth": {"type": "int", "space": "linear", "range": (1, 15)},
    "max_features": {"type": "real", "space": "logit", "range": (0.01, 0.99)},
    "min_samples_split": {"type": "real", "space": "logit", "range": (0.01, 0.99)},
    "min_samples_leaf": {"type": "real", "space": "logit", "range": (0.01, 0.49)},
    "min_weight_fraction_leaf": {"type": "real", "space": "logit", "range": (0.01, 0.49)},
    "min_impurity_decrease": {"type": "real", "space": "linear", "range": (0.0, 0.5)},
}

In particular, I'd like to know how {log, logit, bilog} is determined here. Is there anything that would help me?

I'd love to see an integration of HEBO as baseline right into this benchmark. It seems to be the strongest baseline right now. :)

It is not considered best practice to use install_requires to pin dependencies to specific versions, or to specify sub-dependencies (i.e. dependencies of your dependencies). This is overly-restrictive, and prevents the user from gaining the benefit of dependency upgrades.
https://packaging.python.org/discussions/install-requires-vs-requirements/

Currently, the versions of install_requires and extra_requires are pinned. But it's inconvenience for users. I propose that specifies what a project minimally needs to run correctly.

Hello, just mentioning that the option used for Nevergrad in your plots is OnePlusOne.
As pointed out during early stages of BayesMark, OnePlusOne is certainly not the best tool for your setting. So I would recommend using NGOpt or, even better, compare several options (CMA, NGOpt, CMandAS2, DE, PSO, GeneticDE, cGA).
Have a great 2022!

Allow the user to call experiments with an argument file: commands can be called like
bayesmark-exp --args args.yaml
instead of specifying arguments. All the arguments are simply in the yaml file instead.

Allow the user to use yaml instead of json, if they prefer, for the config.json file.