Transformers are arguably the main workhorse in recent natural language processing
research. By definition, a Transformer is invariant with respect to reordering of the input …

Length generalization, the ability to generalize from small training context sizes to larger
ones, is a critical challenge in the development of Transformer-based language models …

Transformers are emerging as the new workhorse of NLP, showing great success across
tasks. Unlike LSTMs, transformers process input sequences entirely through self-attention …

Transformers have supplanted recurrent models in a large number of NLP tasks. However,
the differences in their abilities to model different syntactic properties remain largely …

Despite their impressive performance in NLP, self-attention networks were recently proved
to be limited for processing formal languages with hierarchical structure, such as $\mathsf …

Neural machine translation (NMT) has proven to be facilitated by curriculum learning which
presents examples in an easy-to-hard order at different training stages. The keys lie in the …

Position encoding (PE), an essential part of self-attention networks (SANs), is used to
preserve the word order information for natural language processing tasks, generating fixed …

Ordered word sequences contain the rich structures that define language. However, it's often
not clear if or how modern pretrained language models utilize these structures. We show …

Transformers are being used extensively across several sequence modeling tasks.
Significant research effort has been devoted to experimentally probe the inner workings of …

Transformer becomes the state-of-the-art translation model, while it is not well studied how
each intermediate component contributes to the model performance, which poses significant …