随机梯度下降算法研究进展
史加荣, 王丹, 尚凡华, 张鹤于 - 自动化学报, 2021 - aas.net.cn
在机器学习领域中, 梯度下降算法是求解最优化问题最重要, 最基础的方法. 随着数据规模的不断
扩大, 传统的梯度下降算法已不能有效地解决大规模机器学习问题. 随机梯度下降算法在迭代 …
扩大, 传统的梯度下降算法已不能有效地解决大规模机器学习问题. 随机梯度下降算法在迭代 …
Lower bounds and optimal algorithms for personalized federated learning
In this work, we consider the optimization formulation of personalized federated learning
recently introduced by Hanzely & Richtarik (2020) which was shown to give an alternative …
recently introduced by Hanzely & Richtarik (2020) which was shown to give an alternative …
Acceleration methods
This monograph covers some recent advances in a range of acceleration techniques
frequently used in convex optimization. We first use quadratic optimization problems to …
frequently used in convex optimization. We first use quadratic optimization problems to …
Pareto invariant risk minimization: Towards mitigating the optimization dilemma in out-of-distribution generalization
Recently, there has been a growing surge of interest in enabling machine learning systems
to generalize well to Out-of-Distribution (OOD) data. Most efforts are devoted to advancing …
to generalize well to Out-of-Distribution (OOD) data. Most efforts are devoted to advancing …
Momentum and stochastic momentum for stochastic gradient, newton, proximal point and subspace descent methods
N Loizou, P Richtárik - Computational Optimization and Applications, 2020 - Springer
In this paper we study several classes of stochastic optimization algorithms enriched with
heavy ball momentum. Among the methods studied are: stochastic gradient descent …
heavy ball momentum. Among the methods studied are: stochastic gradient descent …
Don't jump through hoops and remove those loops: SVRG and Katyusha are better without the outer loop
The stochastic variance-reduced gradient method (SVRG) and its accelerated variant
(Katyusha) have attracted enormous attention in the machine learning community in the last …
(Katyusha) have attracted enormous attention in the machine learning community in the last …
Convex optimization algorithms in medical image reconstruction—in the age of AI
The past decade has seen the rapid growth of model based image reconstruction (MBIR)
algorithms, which are often applications or adaptations of convex optimization algorithms …
algorithms, which are often applications or adaptations of convex optimization algorithms …
Optimal decentralized distributed algorithms for stochastic convex optimization
We consider stochastic convex optimization problems with affine constraints and develop
several methods using either primal or dual approach to solve it. In the primal case, we use …
several methods using either primal or dual approach to solve it. In the primal case, we use …
Learning-rate annealing methods for deep neural networks
Deep neural networks (DNNs) have achieved great success in the last decades. DNN is
optimized using the stochastic gradient descent (SGD) with learning rate annealing that …
optimized using the stochastic gradient descent (SGD) with learning rate annealing that …
A hybrid stochastic optimization framework for composite nonconvex optimization
We introduce a new approach to develop stochastic optimization algorithms for a class of
stochastic composite and possibly nonconvex optimization problems. The main idea is to …
stochastic composite and possibly nonconvex optimization problems. The main idea is to …