Using SuccessiveHalvingPruner about allennlp-optuna HOT 11 CLOSED

Hello @pvcastro, please give me some time to investigate it.
(It should be a bug in allennlp-optuna, sorry for the inconvenience.)

from allennlp-optuna.

@pvcastro Could you please upgrade allennlp-optuna and check if the error still happens?

what exactly is a "resource"

min_resource controls a computation budget for each trial.
The smaller min_resource is, the more likely pruning will occur.

• ref. https://github.com/optuna/optuna/blob/d9a1f83a55953069ce322d0e8d778e91f0def525/optuna/pruners/_successive_halving.py#L182-L186

If you want to learn more detail, please refer to the original paper or ask Optuna dev team on gitter.

from allennlp-optuna.

Hi @himkt !
Leaving "auto" for min_resource is still throwing the same exception after upgrading to 0.1.6.

from allennlp-optuna.

Finally, I found the root cause.

Optuna pruning integration for AllenNLP creates a pruner using environment variables.
If a value None (None object) is specified in config, it would be parsed as "None" (string literal) when constructing a pruner. It causes the problem you encountered.

(Pdb) _get_environment_variables_for_pruner()
{'min_resource': 'None', 'reduction_factor': 4, 'min_early_stopping_rate': 0}

https://github.com/optuna/optuna/blob/master/optuna/integration/allennlp.py#L74-L95

I'll make a PR to avoid this problem.
But for now, could you please specify min_resource to be "auto" explicitly in your configuration?

from allennlp-optuna.

@pvcastro

from allennlp-optuna.

Hi @himkt , as I stated here, leaving "auto" still causes the issue. The "auto" value is being ignored, for some reason.

from allennlp-optuna.

I was doing some debugging, and I think the problem is that at this line at optuna, they don't save the "auto" value, they only save a value if it's an integer. And whey you retrieve the pruner attributes from the environment variables in the method you mentioned, the min_resource is None, and when you try to instantiate a new one with None, it throws the exception.

from allennlp-optuna.

Oh, you're right, thank you.

I now understand the problem is happened due to the mismatch between min_resource and _min_resource.
As you pointed out, SuccessiveHalvingPruner doesn't preserve the min_resource when it is not integer.
I'll discuss how to restore a pruner from attributes such as _min_resource (slightly different from the original min_resource) with Optuna dev team, sorry but please wait. 🙇 (thank you so much for kindly and precise investigations)

from allennlp-optuna.

Sure, glad to help!

from allennlp-optuna.

Sorry for being late @pvcastro. I resolved this problem in optuna/optuna#2731.
Please run pip uninstall optuna and run pip install 'optuna@git+https://github.com/optuna/optuna@master' on your environment. (This fix will be included in the next version of Optuna)

from allennlp-optuna.

Closed as this fix was shipped in Optuna v2.9.0.

from allennlp-optuna.