Improving Generalization Performance by Switching from Adam to SGD

"Improving Generalization Performance by Switching from Adam to SGD" is a research paper that proposes that switching from the Adam optimizer to stochastic gradient descent (SGD) can improve the generalization performance of deep neural networks.

The Adam optimizer is a commonly used optimization algorithm for training neural networks. It combines the advantages of two other optimization algorithms, AdaGrad and RMSProp, to effectively adapt the learning rate during training. However, the authors of the paper argue that Adam's adaptive learning rates can lead to overfitting, where the model performs well on the training data but poorly on new, unseen data.

To address this issue, the paper proposes using SGD instead of Adam during the later stages of training. The authors found that this approach improved the generalization performance of the model on a variety of datasets and tasks.

Overall, the paper suggests that while Adam may be a good optimizer for initial training stages, switching to SGD can help prevent overfitting and ultimately improve the generalization performance of deep neural networks.