Universality laws for gaussian mixtures in generalized linear models

Y Dandi, L Stephan, F Krzakala… - Advances in …, 2024 - proceedings.neurips.cc
A recent line of work in high-dimensional statistics working under the Gaussian mixture
hypothesis has led to a number of results in the context of empirical risk minimization …

Asymptotics of feature learning in two-layer networks after one gradient-step

H Cui, L Pesce, Y Dandi, F Krzakala, YM Lu… - arXiv preprint arXiv …, 2024 - arxiv.org
In this manuscript we investigate the problem of how two-layer neural networks learn
features from data, and improve over the kernel regime, after being trained with a single …

Multinomial logistic regression: Asymptotic normality on null covariates in high-dimensions

K Tan, PC Bellec - Advances in Neural Information …, 2024 - proceedings.neurips.cc
This paper investigates the asymptotic distribution of the maximum-likelihood estimate
(MLE) in multinomial logistic models in the high-dimensional regime where dimension and …

Demystifying disagreement-on-the-line in high dimensions

D Lee, B Moniri, X Huang… - International …, 2023 - proceedings.mlr.press
Evaluating the performance of machine learning models under distribution shifts is
challenging, especially when we only have unlabeled data from the shifted (target) domain …

Precise asymptotic generalization for multiclass classification with overparameterized linear models

D Wu, A Sahai - Advances in Neural Information Processing …, 2024 - proceedings.neurips.cc
We study the asymptotic generalization of an overparameterized linear model for multiclass
classification under the Gaussian covariates bi-level model introduced in Subramanian et …

High-dimensional robust regression under heavy-tailed data: Asymptotics and Universality

U Adomaityte, L Defilippis, B Loureiro… - arXiv preprint arXiv …, 2023 - arxiv.org
We investigate the high-dimensional properties of robust regression estimators in the
presence of heavy-tailed contamination of both the covariates and response functions. In …

Asymptotics of Learning with Deep Structured (Random) Features

D Schröder, D Dmitriev, H Cui, B Loureiro - arXiv preprint arXiv …, 2024 - arxiv.org
For a large class of feature maps we provide a tight asymptotic characterisation of the test
error associated with learning the readout layer, in the high-dimensional limit where the …

Optimal Ridge Regularization for Out-of-Distribution Prediction

P Patil, JH Du, RJ Tibshirani - arXiv preprint arXiv:2404.01233, 2024 - arxiv.org
We study the behavior of optimal ridge regularization and optimal ridge risk for out-of-
distribution prediction, where the test distribution deviates arbitrarily from the train …

Classification of heavy-tailed features in high dimensions: a superstatistical approach

U Adomaityte, G Sicuro, P Vivo - Advances in Neural …, 2024 - proceedings.neurips.cc
We characterise the learning of a mixture of two clouds of data points with generic centroids
via empirical risk minimisation in the high dimensional regime, under the assumptions of …

Sliding down the stairs: how correlated latent variables accelerate learning with neural networks

L Bardone, S Goldt - arXiv preprint arXiv:2404.08602, 2024 - arxiv.org
Neural networks extract features from data using stochastic gradient descent (SGD). In
particular, higher-order input cumulants (HOCs) are crucial for their performance. However …