Transformers have surpassed RNNs in popularity due to their superior abilities in parallel
training and long-term dependency modeling. Recently, there has been a renewed interest …

Recurrent neural networks (RNNs) have fast inference and scale efficiently on long
sequences, but they are difficult to train and hard to scale. We propose Hawk, an RNN with …

Effectively modeling long spatiotemporal sequences is challenging due to the need to model
complex spatial correlations and long-range temporal dependencies simultaneously …

Recent architectural developments have enabled recurrent neural networks (RNNs) to reach
and even surpass the performance of Transformers on certain sequence modeling tasks …

Graph neural networks based on iterative one-hop message passing have been shown to
struggle in harnessing the information from distant nodes effectively. Conversely, graph …

Various linear complexity models, such as Linear Transformer (LinFormer), State Space
Model (SSM), and Linear RNN (LinRNN), have been proposed to replace the conventional …

Few-shot class-incremental learning (FSCIL) confronts the challenge of integrating new
classes into a model with minimal training samples while preserving the knowledge of …

Softmax attention is the principle backbone of foundation models for various artificial
intelligence applications, yet its quadratic complexity in sequence length can limit its …

The strength of modern large-scale neural networks lies in their ability to efficiently adapt to
new tasks with few examples. Although extensive research has investigated the …

Linear Recurrent Neural Networks (LRNNs) such as Mamba, RWKV, GLA, mLSTM, and
DeltaNet have emerged as efficient alternatives to Transformers in large language …