boschresearch / metabo Goto Github PK

View Code? Open in Web Editor NEW

34.0 7.0 11.0 10.15 MB

Meta-Learning Acquisition Functions for Transfer Learning in Bayesian Optimization

Home Page: https://openreview.net/forum?id=ryeYpJSKwr

License: GNU Affero General Public License v3.0

Python 100.00%

bcai paper-resource

metabo's Introduction

MetaBO - Meta-Learning Acquisition Functions for Transfer Learning in Bayesian Optimization

This is the source code accompanying the paper Meta-Learning Acquisition Functions for Transfer Learning in Bayesian Optimization by Volpp et al., ICLR 2020. The paper can be found here. The code allows to reproduce the results from the paper and to train neural acquisition functions on new problems.

Purpose of the project

This software is a research prototype, solely developed for and published as part of the publication cited above. It will neither be maintained nor monitored in any way.

Installation

Clone this repository and run

conda env create -f environment.yml
conda activate metabo

to create and activate a new conda environment named "metabo" with all python packages required to run the experiments.

Run the code

We provide:

Scripts to reproduce the results presented in the paper. These scripts are named evaluate_metabo_<experiment_name>.py. They load pre-trained network weights stored in /metabo/iclr2020/<experiment_name> to reproduce the results without the need of re-training neural acquisition functions. To run these scripts, execute

python evaluate_metabo_<experiment_name>.py

Scripts to re-train the aforementioned neural acquisition functions. These scripts are named train_metabo_<experiment_name>.py. To run these scripts, execute

python train_metabo_<experiment_name>.py

License

"Meta-Learning Acquisition Functions for Transfer Learning in Bayesian Optimization" is open-sourced under the APGL-3.0 license. See the LICENSE file for details.

metabo's People

Contributors

Stargazers

Watchers

Forkers

hyunghunny monirzaman aliaby devpouya qiming-zou mohan-zhang-u jinxcrazy haowencs hadijomaa temesgen-21 rvalenciaaz

metabo's Issues

can the actions be repeated in the eval setting?

could someone please let me know if the actions in the 'svm_hpo' environment are actually stored with the transitions:
transitions = Transition(*zip(*br.memory.copy())) #line number 151; /MetaBO/metabo/eval/evaluate.py

one can see below that actions are being repeated, which causes two confusing questions:

are these actions the index of the hyperparameter configuration being tried?
why is the model repeating the action, as in rerunning the hpo configuration?

when i print out the actions for the 'svm_hpo' environment:
action=(array(117), array(120), array(129), array(123), array(118), array(122), array(121), array(122), array(121), array(121), array(122), array(121), array(121), array(121), array(122), array(121), array(121), array(121), array(122), array(121), array(117), array(120), array(126), array(118), array(119), array(118), array(118), array(118), array(118), array(118), array(118), array(118), array(118), array(118), array(118), array(118), array(118), array(118), array(118), array(118), array(117), array(118), array(119), array(120), array(121), array(122), array(127), array(125), array(128), array(129), array(122), array(127), array(122), array(129), array(127), array(123), array(127), array(122), array(127), array(127), array(117), array(120), array(129), array(119), array(119), array(119), array(119), array(119), array(119), array(119), array(119), array(119), array(119), array(119), array(119), array(119), array(119), array(119), array(119), array(119), array(117), array(119), array(121), array(129), array(118), array(123), array(120), array(119), array(119), array(120), array(119), array(119), array(119), array(119), array(119), array(118), array(119), array(120), array(119), array(119), array(117), array(119), array(122), array(126), array(118), array(121), array(123), array(117), array(117), array(117), array(117), array(117), array(117), array(117), array(117), array(117), array(117), array(117), array(117), array(117), array(117), array(120), array(126), array(118), array(123), array(127), array(117), array(117), array(118), array(117), array(117), array(117), array(117), array(119), array(117), array(117), array(117), array(117), array(117), array(117), array(117), array(120), array(127), array(124), array(123), array(126), array(122), array(124), array(124), array(124), array(124), array(124), array(124), array(124), array(124), array(124), array(124), array(124), array(124), array(124), array(117), array(119), array(127), array(123), array(118), array(128), array(121), array(117), array(117), array(117), array(117), array(117), array(117), array(117), array(117), array(117), array(117), array(117), array(117), array(117), array(117), array(119), array(122), array(127), array(121), array(122), array(123), array(122), array(121), array(122), array(122), array(121), array(122), array(122), array(121), array(122), array(122), array(122), array(121), array(122), array(117), array(118), array(119), array(120), array(121), array(122), array(127), array(129), array(125), array(126), array(128), array(124), array(123), array(117), array(117), array(117), array(117), array(117), array(118), array(117), array(117), array(120), array(127), array(119), array(120), array(120), array(120), array(123), array(120), array(120), array(120), array(120), array(120), array(120), array(120), array(120), array(120), array(120), array(120), array(120), array(117), array(118), array(119), array(120), array(121), array(121), array(121), array(121), array(122), array(121), array(127), array(121), array(121), array(121), array(121), array(121), array(121), array(121), array(121), array(121), array(117), array(122), array(123), array(119), array(118), array(128), array(117), array(117), array(117), array(117), array(117), array(117), array(117), array(117), array(117), array(117), array(117), array(117), array(117), array(117), array(117), array(119), array(120), array(120), array(119), array(120), array(127), array(128), array(120), array(119), array(120), array(119), array(120), array(123), array(119), array(120), array(120), array(119), array(120), array(120))

why is the prepare_data file reading only 2 parameters from the hpo meta datasets?

unfortunately it is not clear as to why in the prepare_data file, only 2 parameters are read:
param_nos_to_extract = [3, 4] for svm meta data
param_nos_to_extract = [0, 1] for adaboost meta data

Could you please comment about this, and also is this related to 'D' in the paper, where it is said that HPO problems are 2D?

evaluation results being deterministic

Hello,

Even when evaluating with different random seeds the results of the transitions are the same for both TAF and Meta-BO algorithms. Is there any other way of setting the seed except changing the env_seed_offset variable in the following:

    # define evaluation run
    eval_spec = {
        "env_id": env_spec["env_id"],
        "env_seed_offset": 100,
        "policy": af,
        "logpath": logpath,
        "load_iter": load_iter,
        "deterministic": deterministic,
        "policy_specs": policy_specs,
        "savepath": savepath,
        "n_workers": n_workers,
        "n_episodes": n_episodes,
        "T": env_spec["T"],
    }

requirement for precomputed gaussian process hyperparameters for test datasets

Dear authors,

Apologies for the opening of too many issues already, unfortunately, there is another query I have regarding the method. I would be grateful for your response.

For the evaluation to work, we need to provide the "gp_hyperparameters.pkl" file. This file has precomputed test dataset gaussian process hyperparameters such as lengthscale, variance and noise variance. These are loaded in the environment as shown below:

           # load gp-hyperparameters
            self.kernel_lengthscale = self.hpo_gp_hyperparameters[dataset]["lengthscale"]
            self.kernel_variance = self.hpo_gp_hyperparameters[dataset]["variance"]
            self.noise_variance = self.hpo_gp_hyperparameters[dataset]["noise_variance"]

I do not understand as to why there is a requirement of a pre-trained GP "model" (and not just hyperparameters) on test dataset beforehand for the methods MetaBO and TAF both to be evaluated on test dataset?

In order to generate these GP hyperparameters ("gp_hyperparameters.pkl" ) I use the following code, where "X" and "y" are from the "objectives.pkl", the meta-data involving hyperparameter configurations and their responses iteratively for each dataset(training and test datasets).

    kernel = GPy.kern.RBF(X.shape[1], ARD=1)
    m = GPy.models.GPRegression(X,y,kernel)
    m.optimize('bfgs', max_iters=200)
    gp_hyperparameters[dataset]={'lengthscale': np.array(m['.*lengthscale']), 'variance': np.array(m['rbf.*variance']), 'noise_variance': np.array(m['.*_noise.variance'])}

Can you also provide feedback as to whether this is how the "gp_hyperparameters.pkl" file was to be generated, because from my perspective this is a trained GP model now and not just hyperparameters of a GP model.

a small mistake in environment.yml

Dear authors,

There is a small mistake in Line 19 in environment.yml:

Missing one "=" in "scikit-learn=0.21.3", should be "scikit-learn==0.21.3"

User can only finish Installation after manually fixing.

I wish you can fix this.