Using mealpy optimizers for training pretrained models about mealpy HOT 4 CLOSED

@thieu1995 Can you please look into this?

from mealpy.

Hi @waseemR02,

Metaheuristic (swarm-based, evolutionary,....) algorithms are gradient-free optimization. Therefor it is completely different than Gradient-based optimization (GD, SGD, Adam, Adagrad,...). Besides, MHAs can't solve a large dimensions problems like deep neural network. You can check several large-scale competition CEC-2023, CEC-2020,.. about evolutionary, they can only handle problem at maximum 1000 dimensions (variables). Deep neural networks (Pre-trained models) are usually a big network with 100k weights or more. You can't really use MHAs to fine-tune this kind of model. In the youtube video, I can make a hybrid model but only with small network.

from mealpy.

Is there any other alternative, as it requires me to use some meta heuristic optimizers?What if we just train some input layers and not the hidden layers(just freeze them). Any help would be appreciated.

from mealpy.

@waseemR02,

I don't know because I'm not familiar with the pre-trained mobilenet-ssd fpn320 model. I haven't used it before. Perhaps you can treat it as a feature selection problem, choosing which hidden layers to freeze or not. You can use the BinaryVar type to achieve this. Let's say you have 10 hidden layers, and you need to select a combination of layers that yields the best results. For example, you could freeze the 3rd and 8th hidden layers. This is a way to apply metaheuristics to your pre-trained model.

from mealpy.