Large language models have been widely adopted but require significant GPU memory for
inference. We develop a procedure for Int8 matrix multiplication for feed-forward and …

As their size increases, Large Languages Models (LLMs) are natural candidates for network
pruning methods: approaches that drop a subset of network weights while striving to …

Quantization methods reduce the number of bits required to represent each parameter in a
model, trading accuracy for smaller memory footprints and inference latencies. However, the …

Length generalization, the ability to generalize from small training context sizes to larger
ones, is a critical challenge in the development of Transformer-based language models …

Transformer architecture has become the fundamental element of the widespread natural
language processing~(NLP) models. With the trends of large NLP models, the increasing …

Similarity measures are a vital tool for understanding how language models represent and
process language. Standard representational similarity measures such as cosine similarity …

Emergent properties have been widely adopted as a term to describe behavior not present
in smaller models but observed in larger models (Wei et al., 2022a). Recent work suggests …

We study the effect of width on the dynamics of feature-learning neural networks across a
variety of architectures and datasets. Early in training, wide neural networks trained on …

Multiple studies have shown that Transformers are remarkably robust to pruning. Contrary to
this received wisdom, we demonstrate that pre-trained Transformer encoders are …

The Transformer architecture aggregates input information through the self-attention
mechanism, but there is no clear understanding of how this information is mixed across the …