This work identifies 18 foundational challenges in assuring the alignment and safety of large
language models (LLMs). These challenges are organized into three different categories …

This paper presents a succinct derivation of the training and generalization performance of a
variety of high-dimensional ridge regression models using the basic tools of random matrix …

On a variety of tasks, the performance of neural networks predictably improves with training
time, dataset size and model size across many orders of magnitude. This phenomenon is …

Second-order optimization has been developed to accelerate the training of deep neural
networks and it is being applied to increasingly larger-scale models. In this study, towards …

Deep learning succeeds by doing hierarchical feature learning, yet tuning Hyper-
Parameters (HP) such as initialization scales, learning rates etc., only give indirect control …

We analyze feature learning in infinite-width neural networks trained with gradient flow
through a self-consistent dynamical field theory. We construct a collection of deterministic …

Recently, there has been growing evidence that if the width and depth of a neural network
are scaled toward the so-called rich feature learning limit ($\mu $ P and its depth extension) …

Deep neural networks (DNNs) exhibit a remarkable ability to automatically learn data
representations, finding appropriate features without human input. Here we present a …

A common theoretical approach to understanding neural networks is to take an infinite-width
limit, at which point the outputs become Gaussian process (GP) distributed. This is known as …

SimSiam is a prominent self-supervised learning method that achieves impressive results in
various vision tasks under static environments. However, it has two critical issues: high …