Which is a common issue with non-convex optimization in gradient descent and a typical mitigation?

• A. Convexity guarantees; mitigation: increase learning rate.
• B. Local minima; mitigation: multiple restarts, stochastic gradient, regularization.
• C. Global minima; mitigation: add noise.
• D. Overfitting; mitigation: collect more data.

Answer: (B)

In non-convex optimization, the search landscape often has many basins of attraction, meaning gradient descent can quickly settle in a local minimum or get stalled at a saddle point, preventing you from finding a good or global solution. The most effective mitigation is to encourage exploration and avoid getting trapped in a single basin. Running multiple restarts with different initializations lets you start the search in different regions of the landscape, increasing the chance of finding a better minimum. Using stochastic gradients introduces noise into the update path, which helps the algorithm escape shallow minima and traverse saddle points that a purely deterministic path might remain stuck in. Regularization also helps by shaping and smoothing the loss surface, promoting simpler models and reducing the severity of problematic local minima. In contrast, assuming convexity isn’t applicable here because the problem isn’t convex, and simply increasing the learning rate can destabilize the optimization. Adding noise to chase a global minimum isn’t a reliable standard fix for non-convex landscapes, and overfitting relates to generalization rather than the optimization trajectory itself.