In recent years, deep neural networks have been successful in both industry and academia,
especially for computer vision tasks. The great success of deep learning is mainly due to its …

At the heart of foundation models is the philosophy of" more is different", exemplified by the
astonishing success in computer vision and natural language processing. However, the …

Vision transformer has achieved competitive performance on a variety of computer vision
applications. However, their storage, run-time memory, and computational demands are …

Abstract Knowledge Distillation (KD) has been validated as an effective model compression
technique for learning compact object detectors. Existing state-of-the-art KD methods for …

Abstract Knowledge distillation transfers the knowledge from a cumbersome teacher to a
small student. Recent results suggest that the student-friendly teacher is more appropriate to …

Knowledge Distillation (KD) is a widely used technique to transfer knowledge from pre-
trained teacher models to (usually more lightweight) student models. However, in certain …

Deep Neural Networks (DNNs), despite their notable progress across information retrieval
tasks, encounter the issues of shortcut learning and struggle with poor generalization due to …

The knowledge of a well-trained deep neural network (aka the “teacher”) is valuable for
learning similar tasks. Knowledge distillation extracts knowledge from the teacher and …

Deep neural networks have achieved remarkable performance for artificial intelligence
tasks. The success behind intelligent systems often relies on large-scale models with high …

Feature distillation makes the student mimic the intermediate features of the teacher. Nearly
all existing feature-distillation methods use L2 distance or its slight variants as the distance …