Although we have witnessed great success of pre-trained models in natural language
processing (NLP) and computer vision (CV), limited progress has been made for general …

Large language models (LLMs) have demonstrated impressive capabilities in natural
language processing. However, their internal mechanisms are still unclear and this lack of …

Widely used alignment techniques, such as reinforcement learning from human feedback
(RLHF), rely on the ability of humans to supervise model behavior-for example, to evaluate …

While the successes of transformers across many domains are indisputable, accurate
understanding of the learning mechanics is still largely lacking. Their capabilities have been …

Whether current or near-term AI systems could be conscious is a topic of scientific interest
and increasing public concern. This report argues for, and exemplifies, a rigorous and …

Self-attention, an architectural motif designed to model long-range interactions in sequential
data, has driven numerous recent breakthroughs in natural language processing and …

Transformer models are revolutionizing machine learning, but their inner workings remain
mysterious. In this work, we present a new visualization technique designed to help …

While alignment algorithms are now commonly used to tune pre-trained language models
towards a user's preferences, we lack explanations for the underlying mechanisms in which …

State Space Models (SSMs) have become serious contenders in the field of sequential
modeling, challenging the dominance of Transformers. At the same time, Mixture of Experts …

Recent large language models have been trained on vast datasets, but also often on
repeated data, either intentionally for the purpose of upweighting higher quality data, or …