LightGBM causes Optuna to go out of memory and crash about mljar-supervised HOT 2 OPEN

Hi @AkshayNovacene,

LightGBM has some issues with memory management microsoft/LightGBM#4239

The solution would be to train each model in separate process, to always release full memory after training. I'm thinking about using Ray framework https://github.com/ray-project/ray for this (you can train each model in separate process and you can distribute training on multiple machines).

from mljar-supervised.

Thanks for the prompt reply. I am still trying to dig into the issue because lightGBM works fine with Optuna when I use Optuna out of the box. I was using the lgb.LGBMClassifier component for training manually. It goes out of memory only when I use the optuna mode through the automl package and try to train Lightgbm.

from mljar-supervised.