Attention mechanism is a central component of the transformer architecture which led to the
phenomenal success of large language models. However, the theoretical principles …

Recently, over-parameterized deep networks, with increasingly more network parameters
than training samples, have dominated the performances of modern machine learning …

Since its inception in" Attention Is All You Need", transformer architecture has led to
revolutionary advancements in NLP. The attention layer within the transformer admits a …

Common explanations for shortcut learning assume that the shortcut improves prediction
only under the training distribution. Thus, models trained in the typical way by minimizing log …

As part of the effort to understand implicit bias of gradient descent in overparametrized
models, several results have shown how the training trajectory on the overparametrized …

Attention mechanism is a central component of the transformer architecture which led to the
phenomenal success of large language models. However, the theoretical principles …

First-order optimization methods tend to inherently favor certain solutions over others when
minimizing a given training objective with multiple local optima. This phenomenon, known …

Adam has become one of the most favored optimizers in deep learning problems. Despite its
success in practice, numerous mysteries persist regarding its theoretical understanding. In …

We study the implicit bias of the general family of steepest descent algorithms, which
includes gradient descent, sign descent and coordinate descent, in deep homogeneous …

In this work, we investigate the margin-maximization bias exhibited by gradient-based
algorithms in classifying linearly separable data. We present an in-depth analysis of the …