Deterministic equivalent and error universality of deep random features learning
This manuscript considers the problem of learning a random Gaussian network function
using a fully connected network with frozen intermediate layers and trainable readout layer …
using a fully connected network with frozen intermediate layers and trainable readout layer …
How two-layer neural networks learn, one (giant) step at a time
We investigate theoretically how the features of a two-layer neural network adapt to the
structure of the target function through a few large batch gradient descent steps, leading to …
structure of the target function through a few large batch gradient descent steps, leading to …
Universality laws for gaussian mixtures in generalized linear models
A recent line of work in high-dimensional statistics working under the Gaussian mixture
hypothesis has led to a number of results in the context of empirical risk minimization …
hypothesis has led to a number of results in the context of empirical risk minimization …
High-dimensional analysis of double descent for linear regression with random projections
F Bach - SIAM Journal on Mathematics of Data Science, 2024 - SIAM
We consider linear regression problems with a varying number of random projections,
where we provably exhibit a double descent curve for a fixed prediction problem, with a high …
where we provably exhibit a double descent curve for a fixed prediction problem, with a high …
A dynamical model of neural scaling laws
On a variety of tasks, the performance of neural networks predictably improves with training
time, dataset size and model size across many orders of magnitude. This phenomenon is …
time, dataset size and model size across many orders of magnitude. This phenomenon is …
On double-descent in uncertainty quantification in overparametrized models
Uncertainty quantification is a central challenge in reliable and trustworthy machine
learning. Naive measures such as last-layer scores are well-known to yield overconfident …
learning. Naive measures such as last-layer scores are well-known to yield overconfident …
Generalized equivalences between subsampling and ridge regularization
We establish precise structural and risk equivalences between subsampling and ridge
regularization for ensemble ridge estimators. Specifically, we prove that linear and quadratic …
regularization for ensemble ridge estimators. Specifically, we prove that linear and quadratic …
[HTML][HTML] An introduction to machine learning: a perspective from statistical physics
A Decelle - Physica A: Statistical Mechanics and its Applications, 2022 - Elsevier
The recent progresses in Machine Learning opened the door to actual applications of
learning algorithms but also to new research directions both in the field of Machine Learning …
learning algorithms but also to new research directions both in the field of Machine Learning …
Subsample ridge ensembles: Equivalences and generalized cross-validation
We study subsampling-based ridge ensembles in the proportional asymptotics regime,
where the feature size grows proportionally with the sample size such that their ratio …
where the feature size grows proportionally with the sample size such that their ratio …
Neural networks: From the perceptron to deep nets
Artificial networks have been studied through the prism of statistical mechanics as
disordered systems since the 1980s, starting from the simple models of Hopfield's …
disordered systems since the 1980s, starting from the simple models of Hopfield's …