While the choice of prior is one of the most critical parts of the Bayesian inference workflow,
recent Bayesian deep learning models have often fallen back on vague priors, such as …

Despite their successes, machine learning techniques are often stochastic, error-prone and
blackbox. How could they then be used in fields such as theoretical physics and pure …

Attention mechanism is a central component of the transformer architecture which led to the
phenomenal success of large language models. However, the theoretical principles …

In deep learning theory, the covariance matrix of the representations serves as aproxy to
examine the network's trainability. Motivated by the success of Transform-ers, we study the …

Attention layers---which map a sequence of inputs to a sequence of outputs---are core
building blocks of the Transformer architecture which has achieved significant …

Self-attention, an architectural motif designed to model long-range interactions in sequential
data, has driven numerous recent breakthroughs in natural language processing and …

It has become standard to solve NLP tasks by fine-tuning pre-trained language models
(LMs), especially in low-data settings. There is minimal theoretical understanding of …

Hyperparameter (HP) tuning in deep learning is an expensive process, prohibitively so for
neural networks (NNs) with billions of parameters. We show that, in the recently discovered …

Hyperparameter (HP) tuning in deep learning is an expensive process, prohibitively so for
neural networks (NNs) with billions of parameters. We show that, in the recently discovered …

In this paper, we conduct a comprehensive study of In-Context Learning (ICL) by addressing
several open questions:(a) What type of ICL estimator is learned by large language …