We explore the impact of parameter sparsity on the scaling behavior of Transformers trained
on massive datasets (ie," foundation models"), in both vision and language domains. In this …

Large Language Models (LLMs) are distinguished by their massive parameter counts, which
typically result in significant redundancy. This work introduces MaskLLM, a learnable …

As Large Language Models (LLMs) have made significant advancements across various
tasks, such as question answering, translation, text summarization, and dialogue systems …

The increasing size of deep neural networks necessitates effective model compression to
improve computational efficiency and reduce their memory footprint. Sparsity and …

N: M sparsity is an emerging model compression method supported by more and more
accelerators to speed up sparse matrix multiplication in deep neural networks. Most existing …

We propose SLoPe, a Double-Pruned Sparse Plus Lazy Low-rank Adapter Pretraining
method for LLMs that improves the accuracy of sparse LLMs while accelerating their …

N: M Structured sparsity has garnered significant interest as a result of relatively modest
overhead and improved efficiency. Additionally, this form of sparsity holds considerable …

To date, 2: 4 sparsity has stood as the only sparse pattern that can be accelerated using
sparse tensor cores on GPUs. In practice, 2: 4 sparsity often possesses low actual speedups …

Model sparsity is a promising approach to reducing parameters or FLOPs of convolutional
neural networks (CNNs). Compared to unstructured or coarse-grained structured sparsity …

Training deep neural networks (DNNs) is costly. Fortunately, Nvidia Ampere and Hopper
GPUs can accelerate matrix multiplications twice as fast as a dense equivalent by …