Abstract Mixture-of-Experts (MoE) has gained increasing popularity as a promising
framework for scaling up large language models (LLMs). However, training MoE from …

Layer-wise distillation is a powerful tool to compress large models (ie teacher models) into
small ones (ie, student models). The student distills knowledge from the teacher by …

Training large, deep neural networks to convergence can be prohibitively expensive. As a
result, often only a small selection of popular, dense models are reused across different …

This survey presents an in-depth exploration of knowledge distillation (KD) techniques
within the realm of Large Language Models (LLMs), spotlighting the pivotal role of KD in …

Large language models (LLMs) have garnered unprecedented advancements across
diverse fields, ranging from natural language processing to computer vision and beyond …

Transformers have emerged as the cornerstone of state-of-the-art natural language
processing models, showcasing exceptional performance across a wide range of AI …

Advancements in LLMs have recently unveiled challenges tied to computational efficiency
and continual scalability due to their requirements of huge parameters, making the …

Abstract All-in-one (AiO) frameworks restore various adverse weather degradations with a
single set of networks jointly. To handle various weather conditions an AiO framework is …

Pre-trained Transformers inherently possess the characteristic of sparse activation, where
only a small fraction of the neurons are activated for each token. While sparse activation has …

Transformer based large language models have achieved tremendous success. However,
the significant memory and computational costs incurred during the inference process make …