Transformers-based models, such as BERT, have been one of the most successful deep
learning models for NLP. Unfortunately, one of their core limitations is the quadratic …

Large transformer models have shown extraordinary success in achieving state-of-the-art
results in many natural language processing applications. However, training and deploying …

Transformers are state-of-the-art models for a variety of sequence modeling tasks. At their
core is an attention function which models pairwise interactions between the inputs at every …

Recent work has shown that either (1) increasing the input length or (2) increasing model
size can improve the performance of Transformer-based neural models. In this paper, we …

Scaling sequence length has become a critical demand in the era of large language models.
However, existing methods struggle with either computational complexity or model …

The quadratic computational and memory complexities of the Transformer's attention
mechanism have limited its scalability for modeling long sequences. In this paper, we …

Transformer-based models have become ubiquitous in natural language processing thanks
to their large capacity, innate parallelism and high performance. The contextualizing …

Deep attention models have advanced the modelling of sequential data across many
domains. For language modelling in particular, the Transformer-XL--a Transformer …

Transformer-based models have achieved state-of-the-art results in many natural language
processing tasks. The self-attention architecture allows transformer to combine information …

The design choices in the Transformer attention mechanism, including weak inductive bias
and quadratic computational complexity, have limited its application for modeling long …