8

In order to converge to the optimum properly, there have been invented different algorithms that use adaptive learning rate, such as AdaGrad, Adam, and RMSProp. On the other hand, there is a learning rate scheduler such as power scheduling and exponential scheduling.

However, I don't understand at what kind of situations you should use one over the other. I feel that using adaptive learning rate optimization algorithm such as Adam is simpler and easier to implement than using learning rate scheduler.

So how can you use it apart properly, depending on what kind of problems?

Blaszard
  • 901
  • 1
  • 13
  • 29
  • Use an adaptive optimizer when you can afford not to wring the last drop of performance, otherwise the schedule becomes yet another hyperparameter to optimize. Adaptiveness is also useful when your data is non-stationary and you have to retrain the model (e.g., in a nightly job). – Emre Aug 16 '17 at 22:32

1 Answers1

4

I'm not sure about other fields but recently in the field of deep neural network training there is this arXiv submission, The Marginal Value of Adaptive Gradient Methods in Machine Learning.

Adaptive optimization methods, which perform local optimization with a metric constructed from the history of iterates, are becoming increasingly popular for training deep neural networks. Examples include AdaGrad, RMSProp, and Adam. We show that for simple overparameterized problems, adaptive methods often find drastically different solutions than gradient descent (GD) or stochastic gradient descent (SGD). We construct an illustrative binary classification problem where the data is linearly separable, GD and SGD achieve zero test error, and AdaGrad, Adam, and RMSProp attain test errors arbitrarily close to half. We additionally study the empirical generalization capability of adaptive methods on several state-of-the-art deep learning models. We observe that the solutions found by adaptive methods generalize worse (often significantly worse) than SGD, even when these solutions have better training performance. These results suggest that practitioners should reconsider the use of adaptive methods to train neural networks.

Emre
  • 10,481
  • 1
  • 29
  • 39
derekhh
  • 211
  • 1
  • 2
  • 4
    While this is informative, this does not answer the original question. Maybe add it as a comment. – Shagun Sodhani Aug 17 '17 at 17:34
  • @ShagunSodhani - I feel like this at least answers the original question partially, i.e. it might be better to train neural networks using these learning rate scheduling based approaches instead of adaptive learning rate optimize algorithms? – derekhh Aug 17 '17 at 20:42
  • 1
    maybe I am missing out something so please correct me. The paper seems to suggest that adaptive methods may not be as good as SGD itself. There is no mention of scheduling based approaches. – Shagun Sodhani Aug 18 '17 at 04:38