If you’ve been working with deep learning for a while, you’re probably well-acquainted with the usual optimizers in PyTorch — SGD, Adam, maybe even AdamW. These are some of the go-to tools in every ML engineer’s toolkit.
But what if I told you that there are pleanty of powerful optimization algorithms out there, which aren’t part of the standard PyTorch package?
Not just that, the algorithms can sometimes outperform Adam for certain tasks and help you crack tough optimization problems you’ve been struggling with!
If that got your attention, great!
In this article, we’ll take a look at some advanced optimization techniques that you may or may not have heard of and see how we can apply them to deep learning.
Specifically, We’ll be talking about Sequential Least Squares ProgrammingSLSQP, Particle Swarm Optimization PSO, Covariant Matrix Adaptation Evolution StrategyCMA-ES, and Simulated Annealing SA.
Why use these algorithms?
There are several key advantages: