An introduction to the mechanics of AutoDiff, exploring its mathematical principles, implementation strategies, and applications in currently most-used frameworks
At the heart of machine learning lies the optimization of loss/objective functions. This optimization process heavily relies on computing gradients of these functions with respect to model parameters. As Baydin et al. (2018) elucidate in their comprehensive survey [1], these gradients guide the iterative updates in optimization algorithms such as stochastic gradient descent (SGD):
θₜ₊₁ = θₜ – α ∇θ L(θₜ)
Where:
- θₜ represents the model parameters at step t
- α is the learning rate
- ∇_θ L(θₜ) denotes the gradient of the loss function L with respect to the parameters θ
This simple update rule belies the complexity of computing gradients in deep neural networks with millions or even billions of parameters.