We consider variants of trust-region and adaptive cubic regularization methods for non-
convex optimization, in which the Hessian matrix is approximated. Under certain condition …

As more computational resources become available, machine learning researchers train
ever larger neural networks on millions of data points using stochastic gradient descent …

For solving large-scale non-convex problems, we propose inexact variants of trust region
and adaptive cubic regularization methods, which, to increase efficiency, incorporate various …

For solving large-scale nonconvex problems, we propose inexact variants of trust region and
adaptive cubic regularization methods, which, to increase efficiency, incorporate various …

Deep learning involves a difficult nonconvex optimization problem with a large number of
weights between any two adjacent layers of a deep structure. To handle large data sets or …

Second-order methods for neural network optimization have several advantages over
methods based on first-order gradient descent, including better scaling to large mini-batch …

Deep learning involves a difficult non-convex optimization problem, which is often solved by
stochastic gradient (SG) methods. While SG is usually effective, it may not be robust in some …

This work presents a new automatic differentiation method, Nilpotent Matrix Differentiation
(NMD), capable of propagating any order of mixed or univariate derivative through common …

Hessian-based analysis/computation is widely used in scientific computing. However, due to
the (incorrect, but in our experience widespread) belief that Hessian-based computations …

Deep learning involves a difficult non-convex optimization problem, which is often solved by
stochastic gradient (SG) methods. While SG is usually effective, it may not be robust in some …