Despite the practical success of deep neural networks, a comprehensive theoretical
framework that can predict practically relevant scores, such as the test accuracy, from …

We present a simple picture of the training process of self-supervised learning methods with
dual deep networks. In our picture, these methods learn their high-dimensional embeddings …

We study the effect of width on the dynamics of feature-learning neural networks across a
variety of architectures and datasets. Early in training, wide neural networks trained on …

Understanding how neural networks learn features, or relevant patterns in data, for
prediction is necessary for their reliable use in technological and scientific applications. In …

The push to train ever larger neural networks has motivated the study of initialization and
training at large network width. A key challenge is to scale training so that a network's …

Understanding the mechanism of how convolutional neural networks learn features from
image data is a fundamental problem in machine learning and computer vision. In this work …

On a variety of tasks, the performance of neural networks predictably improves with training
time, dataset size and model size across many orders of magnitude. This phenomenon is …

We perform a study on the generalization ability of the wide two-layer ReLU neural network
on $\mathbb {R} $. We first establish some spectral properties of the neural tangent kernel …

Humans and animals routinely infer relations between different items or events and
generalize these relations to novel combinations of items. This allows them to respond …

Decades-long literature testifies to the success of statistical mechanics at clarifying
fundamental aspects of deep learning. Yet the ultimate goal remains elusive: we lack a …