Comments (6)
Thanks for opening this issue! I'm afraid I'm not understanding your question. Could you please be a bit more specific, or post some code to illustrate what you want to do?
Regarding validation predictions, all of the repo's examples show how HH creates validation sets automatically for you via the Environment
kwargs cv_type
and cv_params
. HH also automatically makes predictions for the out-of-fold (OOF) datasets and evaluates those predictions, which you can see during Experiment logging. At the end of the Experiment/OptPro, all results (including OOF/Holdout/Test predictions) are automatically saved in the directory given to Environment
's results_path
, so you can find them there.
If you're wondering about Holdout/Test datasets, rather than validation, the holdout_test_datasets_example
should be helpful.
For saving models, each library has different methods of doing this, but you can make a custom lambda_callback
to save your models. Here's a simple lambda_callback_example
.
As far as using your saved Experiments, their results are all stored in the directory above, so you're free to use them however you would normally use your results: Ensembling, averaging predictions, checking the Leaderboard to compare performance, etc.
Sorry I'm not quite getting your question. Please let me know if I missed anything, and thanks again for asking!
from hyperparameter_hunter.
For saving models, each library has different methods of doing this, but you can make a custom lambda_callback to save your models. Here's a simple lambda_callback_example.
Thats it. So I can just write a function for saving the best model and use it together with a lambda_callback (on_run_end?) and dumb my model with joblib.
Thanks for your help.
from hyperparameter_hunter.
@labarababa, I've just pushed an example detailing how to make a lambda_callback
for model-saving.
You can find the new example in PR #198. The broken Travis build is due to an unrelated issue.
Would you mind checking out the new example, and letting me know if that helps answer your question?
from hyperparameter_hunter.
Yup, this solves the problem and its understandable und very detailed
Very good addtition to the examples as well.
Thank you for your efforts.
from hyperparameter_hunter.
Thanks for the great suggestion! I'll close this issue once it's merged. If you have any other questions or ideas, I'd love to hear them! Thanks for your time!
from hyperparameter_hunter.
Closed by #198
from hyperparameter_hunter.
Related Issues (20)
- Support for various advanced functionality? HOT 9
- Unable to access docs HOT 2
- How to do predict_proba in catboost classifier? HOT 3
- Support for nested parameters/parameterizing objects that can't be called by name. HOT 2
- How to handle variable number of layers?
- How can I set class weights in a multiclass classification with imbalance dataset? HOT 1
- OSError: could not get source code HOT 9
- Q: what is RandomForestOptPro exactly? HOT 3
- Doc improvement suggestion HOT 4
- logloss issue with multiclass task HOT 4
- Metaclass conflict with Keras 2.3.0 HOT 1
- Timeseries Cross Validation HOT 6
- Any way to send hyperparameter_hunter.Integer to custom feature_engineer function? HOT 3
- pd.DataFrame.sparse supported in environment?
- Interface with pytorch or tensorflow models HOT 2
- Problem with library scikit-optimize in Python 3.6 HOT 9
- ROC AUC scores don't match to those of sklearn HOT 1
- Failed to import packages from hyperparameter_hunter HOT 3
- ImportError: cannot import name 'Log10' HOT 1
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from hyperparameter_hunter.