A collection of multi-fidelity benchmarks with first class support for user priors
Home Page: https://automl.github.io/mf-prior-bench/
License: Apache License 2.0
Please check out the docs automl.github.io/mf-prior-bench!
This causes some serialization errors with optimizers
In my opinion, having the same function signatures for the two types of retrieval makes sense:
query(config: C, fidelity) -> R: gives the point or the exact result at fidelitytrajectory(config: C, fidelity) -> List[R]: gives the full learning curve until fidelityAdditionally, I was wondering if we can already have a placeholder parameter that can be further passed to either a benchmark state during initialisation or to these function calls above. That is whether the evaluation is being continued or thawed.
The only thing that should change then is the cost of continuations. Not sure where then is the best place to change. We could also kind of do it post-hoc for the evaluations made by subtracting the cost incurred for a lower fidelity evaluation.
For the non-benchmark case, NePS anyways handles continuation so should work out fine I believe.
@DaStoll @eddiebergman thoughts?
Please feel free to take the call.
I installed the libary using pypi (pip install pip install mf-prior-bench) and I am working with the PD1 model lm1b_transformer_2048.
When executing the follwoing basic example
import mfpbench
benchmark = mfpbench.get("lm1b_transformer_2048") # example pd1 benchmark
# This example is based on https://github.com/automl/mf-prior-bench/blob/main/docs/quickstart.md
print(benchmark.name)# There is a list of attributes accessible from the benchmark object
config = benchmark.sample(n = 1, seed=0)[0]
print(config)
result = benchmark.query(config)
print(result)the following warnings are raised
ARNING: ../src/learner.cc:888: Found JSON model saved before XGBoost 1.6, please save the model using current version again. The support for old JSON model will be discontinued in XGBoost 2.3.
[10:38:01] WARNING: ../src/learner.cc:888: Found JSON model saved before XGBoost 1.6, please save the model using current version again. The support for old JSON model will be discontinued in XGBoost 2.3.
/home/lukas/anaconda3/envs/automatic_stopping/lib/python3.9/site-packages/xgboost/data.py:312: FutureWarning: is_sparse is deprecated and will be removed in a future version. Check `isinstance(dtype, pd.SparseDtype)` instead.
if is_sparse(dtype):
/home/lukas/anaconda3/envs/automatic_stopping/lib/python3.9/site-packages/xgboost/data.py:314: FutureWarning: is_categorical_dtype is deprecated and will be removed in a future version. Use isinstance(dtype, CategoricalDtype) instead
elif is_categorical_dtype(dtype) and enable_categorical:
/home/lukas/anaconda3/envs/automatic_stopping/lib/python3.9/site-packages/xgboost/data.py:345: FutureWarning: is_categorical_dtype is deprecated and will be removed in a future version. Use isinstance(dtype, CategoricalDtype) instead
if is_categorical_dtype(dtype)
/home/lukas/anaconda3/envs/automatic_stopping/lib/python3.9/site-packages/xgboost/data.py:336: FutureWarning: is_categorical_dtype is deprecated and will be removed in a future version. Use isinstance(dtype, CategoricalDtype) instead
return is_int or is_bool or is_float or is_categorical_dtype(dtype)
/home/lukas/anaconda3/envs/automatic_stopping/lib/python3.9/site-packages/xgboost/data.py:312: FutureWarning: is_sparse is deprecated and will be removed in a future version. Check `isinstance(dtype, pd.SparseDtype)` instead.
if is_sparse(dtype):
/home/lukas/anaconda3/envs/automatic_stopping/lib/python3.9/site-packages/xgboost/data.py:314: FutureWarning: is_categorical_dtype is deprecated and will be removed in a future version. Use isinstance(dtype, CategoricalDtype) instead
elif is_categorical_dtype(dtype) and enable_categorical:
/home/lukas/anaconda3/envs/automatic_stopping/lib/python3.9/site-packages/xgboost/data.py:345: FutureWarning: is_categorical_dtype is deprecated and will be removed in a future version. Use isinstance(dtype, CategoricalDtype) instead
if is_categorical_dtype(dtype)
/home/lukas/anaconda3/envs/automatic_stopping/lib/python3.9/site-packages/xgboost/data.py:336: FutureWarning: is_categorical_dtype is deprecated and will be removed in a future version. Use isinstance(dtype, CategoricalDtype) instead
return is_int or is_bool or is_float or is_categorical_dtype(dtype)
Seems to overwrite it everytime there's a push, might be something to do with --force, or the aliases. Likely need to test this out locally.
Testing out locally with mike means it will just update the branch gh-pages which you should also have pulled locally.
The correct way to download datasets is python -m mfpbench download [--data-dir]. The documentation and error messages need to be updated.
This will be done on the next update for our currently running experiments. I don't want to needlessly modify things right now and potentially cause any issues.
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
Django
The Web framework for perfectionists with deadlines.
Laravel
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
Microsoft
Open source projects and samples from Microsoft.
Google
Google ❤️ Open Source for everyone.
Alibaba
Alibaba Open Source for everyone
D3
Data-Driven Documents codes.
Tencent
China tencent open source team.