Deterministic equivalent and error universality of deep random features learning

D Schröder, H Cui, D Dmitriev… - … on Machine Learning, 2023 - proceedings.mlr.press
This manuscript considers the problem of learning a random Gaussian network function
using a fully connected network with frozen intermediate layers and trainable readout layer …

How two-layer neural networks learn, one (giant) step at a time

Y Dandi, F Krzakala, B Loureiro, L Pesce… - arXiv preprint arXiv …, 2023 - arxiv.org
We investigate theoretically how the features of a two-layer neural network adapt to the
structure of the target function through a few large batch gradient descent steps, leading to …

Universality laws for gaussian mixtures in generalized linear models

Y Dandi, L Stephan, F Krzakala… - Advances in …, 2024 - proceedings.neurips.cc
A recent line of work in high-dimensional statistics working under the Gaussian mixture
hypothesis has led to a number of results in the context of empirical risk minimization …

High-dimensional analysis of double descent for linear regression with random projections

F Bach - SIAM Journal on Mathematics of Data Science, 2024 - SIAM
We consider linear regression problems with a varying number of random projections,
where we provably exhibit a double descent curve for a fixed prediction problem, with a high …

A dynamical model of neural scaling laws

B Bordelon, A Atanasov, C Pehlevan - arXiv preprint arXiv:2402.01092, 2024 - arxiv.org
On a variety of tasks, the performance of neural networks predictably improves with training
time, dataset size and model size across many orders of magnitude. This phenomenon is …

On double-descent in uncertainty quantification in overparametrized models

L Clarté, B Loureiro, F Krzakala… - International …, 2023 - proceedings.mlr.press
Uncertainty quantification is a central challenge in reliable and trustworthy machine
learning. Naive measures such as last-layer scores are well-known to yield overconfident …

Generalized equivalences between subsampling and ridge regularization

P Patil, JH Du - Advances in Neural Information Processing …, 2024 - proceedings.neurips.cc
We establish precise structural and risk equivalences between subsampling and ridge
regularization for ensemble ridge estimators. Specifically, we prove that linear and quadratic …

[HTML][HTML] An introduction to machine learning: a perspective from statistical physics

A Decelle - Physica A: Statistical Mechanics and its Applications, 2022 - Elsevier
The recent progresses in Machine Learning opened the door to actual applications of
learning algorithms but also to new research directions both in the field of Machine Learning …

Subsample ridge ensembles: Equivalences and generalized cross-validation

JH Du, P Patil, AK Kuchibhotla - arXiv preprint arXiv:2304.13016, 2023 - arxiv.org
We study subsampling-based ridge ensembles in the proportional asymptotics regime,
where the feature size grows proportionally with the sample size such that their ratio …

Neural networks: From the perceptron to deep nets

M Gabrié, S Ganguli, C Lucibello… - Spin Glass Theory and …, 2023 - World Scientific
Artificial networks have been studied through the prism of statistical mechanics as
disordered systems since the 1980s, starting from the simple models of Hopfield's …