Support for nested parameters/parameterizing objects that can't be called by name. about hyperparameter_hunter HOT 2 OPEN

Thanks for raising this!

You’re absolutely right. Currently, we don’t support optimizing both a Keras optimizer object and its parameters at the same time. As you say, optimizing nested parameters in general currently isn’t supported. However the Keras optimizers are specifically problematic because they each have different parameters.

While all optimizers do have an lr kwarg, some have other kwargs that different optimizers don’t have. For example, an attempt to search through optimizers “sgd” and “rmsprop”, as well as momentum gets weird because RMSprop doesn’t have a momentum kwarg. Furthermore, many optimizers have different default lr values.

As you mentioned, this could probably be addressed by adding support for nested/conditional hyperparameter optimization, but I still think it gets messy rather quickly in the case of Keras optimizers. This messiness is compounded when we consider that HH is going to automatically read all of our old Experiments and try to match them with the new search space to jump-start optimization. All the old Experiment results, of course, will also have the default values for hyperparameters that weren’t explicitly given.

All that said, I would very much like to support nested/conditional dimensions…

Could you please clarify what you mean by,

This way i can return optimizers that can’t be called by name (Ie. Adamax or Nadam)

If you’re referring to using Categorical on the string names of optimizers, then “adamax” and “nadam” do work; however, I think I may have misunderstood.

…

The problem is i don’t think you can call any non-native function inside your build_fn…

That’s correct. I won’t bore you with all the technical details, but HH actually rewrites the Keras build_fn its given before evaluating it, and this currently leaves out any non-native or non-Keras stuff, which is the reason for the NameError you’re getting.

Doing this may seem unnecessarily complicated, but the reason for it is to enable providing search dimensions directly to Keras layers. Otherwise it wouldn’t be possible to do something like Dense(Integer(50, 150)) because Dense is expecting a native Python int, not an Integer instance, as its first argument.

If you have any ideas or would actually like some more technical details, I’d be more than happy to discuss it further, but that’s the “short” answer to why you’re getting that NameError.

Your wrapper function approach does seem really interesting, though, and I’m trying to figure out how we could integrate it into HH to support nested/conditional dimensions.

from hyperparameter_hunter.

If you’re referring to using Categorical on the string names of optimizers, then “adamax” and “nadam” do work; however, I think I may have misunderstood.

adamax and nadam are callable by name? I wasn't aware. Either way i think we'd have the same problem when using custom optimizers like adabound, that really have to be defined in a wrapper function.

Your wrapper function approach does seem really interesting, though, and I’m trying to figure out how we could integrate it into HH to support nested/conditional dimensions.

Maybe it is possible to allow a list of methods as params in the build_fn function? Then if we annotated our wrappers HH could possibly supply the appropriate function by name. This way we can supply our own wrapper functions. I'm probably over simplifying things...

from hyperparameter_hunter.