Abstract Knowledge distillation (KD) is an effective training strategy to improve the
lightweight student models under the guidance of cumbersome teachers. However, the large …

In this paper, we present Auto-KD, the first automated search framework for optimal
knowledge distillation design. Traditional distillation techniques typically require handcrafted …

Abstract Knowledge distillation (KD) has emerged as an effective technique for compressing
models that can enhance the lightweight model. Conventional KD methods propose various …

Abstract Mixed-Precision Quantization (MQ) can achieve a competitive accuracy-complexity
trade-off for models. Conventional training-based search methods require time-consuming …

In this paper, we introduce Auto-GAS, the first training-free Generative Architecture Search
(GAS) framework enabled by an auto-discovered proxy. Generative models like Generative …

In this paper, we present AttnZero, the first framework for automatically discovering efficient
attention modules tailored for Vision Transformers (ViTs). While traditional self-attention in …

Neural architecture search (NAS) has emerged as one successful technique to find robust
deep neural network (DNN) architectures. However, most existing robustness evaluations in …

Large language models (LLMs) containing tens of billions of parameters (or even more)
have demonstrated impressive capabilities in various NLP tasks. However, substantial …

Abstract Knowledge distillation improves student model performance. However, using a
larger teacher model does not necessarily result in better distillation gains due to significant …

Building efficient neural network architectures can be a time-consuming task requiring
extensive expert knowledge. This task becomes particularly challenging for edge devices …