A long-standing problem in stochastic optimization is proving that\rsgd, the without-
replacement version of\sgd, converges faster than the usual with-replacement\sgd. Building …

Nonconvex optimization is central in solving many machine learning problems, in which
block-wise structure is commonly encountered. In this work, we propose cyclic block …

Stochastic gradient descent without replacement sampling is widely used in practice for
model training. However, the vast majority of SGD analyses assumes data is sampled with …

Optimization techniques are at the core of data science, including data analysis and
machine learning. An understanding of basic optimization techniques and their fundamental …

Block coordinate descent (BCD) methods are widely used for large-scale numerical
optimization because of their cheap iteration costs, low memory requirements, amenability to …

Optimization is among the richest modeling languages in science. In statistics and machine
learning, for instance, inference is typically posed as an optimization problem. While there …

This paper concerns the worst-case complexity of cyclic coordinate descent (C-CD) for
minimizing a convex quadratic function, which is equivalent to Gauss–Seidel method …

We consider first-order methods with constant step size for minimizing locally Lipschitz
coercive functions that are tame in an o-minimal structure on the real field. We prove that if …

Exploiting partial first-order information in a cyclic way is arguably the most natural strategy
to obtain scalable first-order methods. However, despite their wide use in practice, cyclic …

Semidefinite programming (SDP) with diagonal constraints arise in many optimization
problems, such as Max-Cut, community detection and group synchronization. Although …