Optimization is a critical component in deep learning. We think optimization for neural
networks is an interesting topic for theoretical research due to various reasons. First, its …

One of the major concerns for neural network training is that the nonconvexity of the
associated loss functions may cause a bad landscape. The recent success of neural …

When and why can a neural network be successfully trained? This article provides an
overview of optimization algorithms and theory for training neural networks. First, we discuss …

In this paper, online deep learning (DL)-based channel estimation algorithm for doubly
selective fading channels is proposed by employing the deep neural network (DNN). With …

Understanding the implicit bias of Stochastic Gradient Descent (SGD) is one of the key
challenges in deep learning, especially for overparametrized models, where the local …

Artificial intelligence (AI) based downlink channel state information (CSI) prediction for
frequency division duplexing (FDD) massive multiple-input multiple-output (MIMO) systems …

Recent work has revealed many intriguing empirical phenomena in neural network training,
despite the poorly understood and highly complex loss landscapes and training dynamics …

Motivated by recent increased interest in optimization algorithms for non-convex
optimization in application to training deep neural networks and other optimization problems …

This paper shows that every sublevel set of the loss function of a class of deep over-
parameterized neural nets with piecewise linear activation functions is connected and …

Mode connectivity is a surprising phenomenon in the loss landscape of deep nets. Optima---
at least those discovered by gradient-based optimization---turn out to be connected by …