Recent years have seen a phenomenal rise in the performance and applications of
transformer neural networks. The family of transformer networks, including Bidirectional …

Recent advances in state-of-the-art DNN architecture design have been moving toward
Transformer models. These models achieve superior accuracy across a wide range of …

Transformer model is becoming prevalent in various AI applications with its outstanding
performance. However, the high cost of computation and memory footprint make its …

We focus on addressing the dense backward propagation issue for training efficiency of N:
M fine-grained sparsity that preserves at most N out of M consecutive weights and achieves …

Transformers have become the state-of-the-art architectures for various tasks in Natural
Language Processing (NLP) and Computer Vision (CV); however, their space and …

Masked Autoencoders (MAE) and its variants have proven to be effective for pretraining
large-scale Vision Transformers (ViTs). However, small-scale models do not benefit from the …

Recent innovations on hardware (eg Nvidia A100) have motivated learning N: M structured
sparsity masks from scratch for fast model inference. However, state-of-the-art learning …

Quantization can accelerate large language model (LLM) inference. Going beyond INT8
quantization, the research community is actively exploring even lower precision, such as …

Ubiquitous chemicals in the environment may pose a threat to human health and the
ecosystem, so comprehensive toxicity information must be obtained. Due to the inability of …

Pre-trained BERT models have achieved impressive accuracy on natural language
processing (NLP) tasks. However, their excessive amount of parameters hinders them from …