How to efficiently serve ever-larger trained natural language models in practice has become
exceptionally challenging even for powerful cloud servers due to their prohibitive …

The growing size of neural language models has led to increased attention in model
compression. The two predominant approaches are pruning, which gradually removes …

Pruning is an effective way to reduce the huge inference cost of Transformer models.
However, prior work on pruning Transformers requires retraining the models. This can add …

Transformer models have demonstrated their promising potential and achieved excellent
performance on a series of computer vision tasks. However, the huge computational cost of …

Efficient deployment of transformer models in practice is challenging due to their inference
cost including memory footprint, latency, and power consumption, which scales quadratically …

Extreme compression, particularly ultra-low bit precision (binary/ternary) quantization, has
been proposed to fit large NLP models on resource-constraint devices. However, to …

Abstract Large Language Models (LLMs) have achieved great success in solving difficult
tasks across many domains, but such success comes with a high computation cost, and …

Ad relevance modeling plays a critical role in online advertising systems including Microsoft
Bing. To leverage powerful transformers like BERT in this low-latency setting, many existing …

Abstract Recent advancements in Scene Text Visual Question Answering (Text-VQA)
employ autoregressive Transformers, showing improved performance with larger models …

As language models increase in size by the day, methods for efficient inference are critical to
leveraging their capabilities for various applications. Prior work has investigated techniques …