This monograph covers some recent advances in a range of acceleration techniques
frequently used in convex optimization. We first use quadratic optimization problems to …

Can we accelerate the convergence of gradient descent without changing the algorithm—
just by judiciously choosing stepsizes? Surprisingly, we show that the answer is yes. Our …

Momentum-based acceleration of stochastic gradient descent (SGD) is widely used in deep
learning. We propose the quasi-hyperbolic momentum algorithm (QHM) as an extremely …

The use of momentum in stochastic gradient methods has become a widespread practice in
machine learning. Different variants of momentum, including heavy-ball momentum …

We propose the first global accelerated gradient method for Riemannian manifolds. Toward
establishing our results, we revisit Nesterov's estimate sequence technique and develop a …

In this paper, we develop a unified framework capable of certifying both exponential and
subexponential convergence rates for a wide range of iterative first-order optimization …

We present the Branch-and-Bound Performance Estimation Programming (BnB-PEP), a
unified methodology for constructing optimal first-order methods for convex and nonconvex …

We present an optimal gradient method for smooth strongly convex optimization. The
method is optimal in the sense that its worst-case bound on the distance to an optimal point …

Simple stochastic momentum methods are widely used in machine learning optimization,
but their good practical performance is at odds with an absence of theoretical guarantees of …

This paper presents a new class of robust zero-order algorithms for the solution of real-time
optimization problems with acceleration. In particular, we propose a family of extremum …