In the age of artificial intelligence, the best approach to handling huge amounts of data is a
tremendously motivating and hard problem. Among machine learning models, stochastic …

The recently proposed stochastic Polyak stepsize (SPS) and stochastic line-search (SLS) for
SGD have shown remarkable effectiveness when training over-parameterized models …

Very little is known about the training dynamics of adaptive gradient methods like Adam in
deep learning. In this paper, we shed light on the behavior of these algorithms in the full …

Stochastic gradient descent (SGD) is one of the most popular algorithms in modern machine
learning. The noise encountered in these applications is different from that in many …

Abstract Recently Loizou et al.(2021), proposed and analyzed stochastic gradient descent
(SGD) with stochastic Polyak stepsize (SPS). The proposed SPS comes with strong …

Deep Learning algorithms have achieved a great success in many domains where large
scale datasets are used. However, training these algorithms on high dimensional data …

Adaptive algorithms like AdaGrad and AMSGrad are successful in nonconvex optimization
owing to their parameter-agnostic ability–requiring no a priori knowledge about problem …

We prove that the iterates produced by, either the scalar step size variant, or the
coordinatewise variant of AdaGrad algorithm, are convergent sequences when applied to …

The presence of outliers can potentially significantly skew the parameters of machine
learning models trained via stochastic gradient descent (SGD). In this paper we propose a …

In this paper, we investigate the problem of stochastic multi-level compositional optimization,
where the objective function is a composition of multiple smooth but possibly non-convex …