Fine-tuning visual models has been widely shown promising performance on many
downstream visual tasks. With the surprising development of pre-trained visual foundation …

Unlike existing knowledge distillation methods focus on the baseline settings, where the
teacher models and training strategies are not that strong and competing as state-of-the-art …

Deep learning-based intelligent vehicle perception has been developing prominently in
recent years to provide a reliable source for motion planning and decision making in …

In this paper, we present Auto-KD, the first automated search framework for optimal
knowledge distillation design. Traditional distillation techniques typically require handcrafted …

The representation gap between teacher and student is an emerging topic in knowledge
distillation (KD). To reduce the gap and improve the performance, current methods often …

Abstract Knowledge distillation (KD) has emerged as an effective technique for compressing
models that can enhance the lightweight model. Conventional KD methods propose various …

Existing feature distillation methods commonly adopt the One-to-one Representation
Matching between any pre-selected teacher-student layer pair. In this paper, we present N …

Large pre-trained transformers are on top of contemporary semantic segmentation
benchmarks, but come with high computational cost and a lengthy training. To lift this …

Anatomical consistency in biomarker segmentation is crucial for many medical image
analysis tasks. A promising paradigm for achieving anatomically consistent segmentation …

Abstract Unlike the conventional Knowledge Distillation (KD), Self-KD allows a network to
learn knowledge from itself without any guidance from extra networks. This paper proposes …