Algorithmic reasoning requires capabilities which are most naturally understood through
recurrent models of computation, like the Turing machine. However, Transformer models …

Vision transformers (ViTs) have pushed the state-of-the-art for various visual recognition
tasks by patch-wise image tokenization followed by self-attention. However, the employment …

Transformers with linear attention allow for efficient parallel training but can simultaneously
be formulated as an RNN with 2D (matrix-valued) hidden states, thus enjoying linear (with …

In recent computer vision research, the advent of the Vision Transformer (ViT) has rapidly
revolutionized various architectural design efforts: ViT achieved state-of-the-art image …

Performing inference on large volumes of samples with large language models (LLMs) can
be computationally and financially costly in industry and real-world use. We propose batch …

This work provides empirical evidence that Mamba, a newly proposed selective structured
state space model, has similar in-context learning (ICL) capabilities as transformers. We …

Though nearest neighbor Machine Translation ($ k $ NN-MT)\citep {khandelwal2020nearest
} has proved to introduce significant performance boosts over standard neural MT systems, it …

The attention mechanism is considered the backbone of the widely-used Transformer
architecture. It contextualizes the input by computing input-specific attention matrices. We …

Natural language processing researchers have identified limitations of evaluation
methodology for generation tasks, with new questions raised about the validity of automatic …

Recent work has shown that attention-based language models excel at recall, the ability to
ground generations in tokens previously seen in context. However, the efficiency of attention …