Vision transformers using self-attention or its proposed alternatives have demonstrated
promising results in many image related tasks. However, the underpinning inductive bias of …

We develop fast algorithms and robust software for convex optimization of two-layer neural
networks with ReLU activation functions. Our work leverages a convex re-formulation of the …

Understanding the fundamental mechanism behind the success of deep neural networks is
one of the key challenges in the modern machine learning literature. Despite numerous …

We describe the convex semi-infinite dual of the two-layer vector-output ReLU neural
network training problem. This semi-infinite dual admits a finite dimensional representation …

Batch Normalization (BN) is a commonly used technique to accelerate and stabilize training
of deep neural networks. Despite its empirical success, a full theoretical understanding of …

We propose a novel theoretical framework of analysis for Generative Adversarial Networks
(GANs). We reveal a fundamental flaw of previous analyses which, by incorrectly modeling …

Understanding the fundamental principles behind the success of deep neural networks is
one of the most important open questions in the current literature. To this end, we study the …

Threshold activation functions are highly preferable in neural networks due to their efficiency
in hardware implementations. Moreover, their mode of operation is more interpretable and …

Recently, theoretical analyses of deep neural networks have broadly focused on two
directions: 1) Providing insight into neural network training by SGD in the limit of infinite …

Training deep neural networks is a challenging non-convex optimization problem. Recent
work has proven that the strong duality holds (which means zero duality gap) for regularized …