Knowledge distillation and student-teacher learning for visual intelligence: A review and new outlooks

L Wang, KJ Yoon - IEEE transactions on pattern analysis and …, 2021 - ieeexplore.ieee.org
Deep neural models, in recent years, have been successful in almost every field, even
solving the most complex problem statements. However, these models are huge in size with …

Factorizing knowledge in neural networks

X Yang, J Ye, X Wang - European Conference on Computer Vision, 2022 - Springer
In this paper, we explore a novel and ambitious knowledge-transfer task, termed Knowledge
Factorization (KF). The core idea of KF lies in the modularization and assemblability of …

Diffusion probabilistic model made slim

X Yang, D Zhou, J Feng… - Proceedings of the IEEE …, 2023 - openaccess.thecvf.com
Despite the visually-pleasing results achieved, the massive computational cost has been a
long-standing flaw for diffusion probabilistic models (DPMs), which, in turn, greatly limits …

[PDF][PDF] 知识蒸馏研究综述

黄震华, 杨顺志, 林威, 倪娟, 孙圣力, 陈运文, 汤庸 - 计算机学报, 2022 - 159.226.43.17
摘要高性能的深度学习网络通常是计算型和参数密集型的, 难以应用于资源受限的边缘设备.
为了能够在低资源设备上运行深度学习模型, 需要研发高效的小规模网络 …

Knowledge distillation: A survey

J Gou, B Yu, SJ Maybank, D Tao - International Journal of Computer Vision, 2021 - Springer
In recent years, deep neural networks have been successful in both industry and academia,
especially for computer vision tasks. The great success of deep learning is mainly due to its …

Curriculum temperature for knowledge distillation

Z Li, X Li, L Yang, B Zhao, R Song, L Luo, J Li… - Proceedings of the …, 2023 - ojs.aaai.org
Most existing distillation methods ignore the flexible role of the temperature in the loss
function and fix it as a hyper-parameter that can be decided by an inefficient grid search. In …

WaveNet: Wavelet network with knowledge distillation for RGB-T salient object detection

W Zhou, F Sun, Q Jiang, R Cong… - IEEE Transactions on …, 2023 - ieeexplore.ieee.org
In recent years, various neural network architectures for computer vision have been devised,
such as the visual transformer and multilayer perceptron (MLP). A transformer based on an …

Distilling knowledge from graph convolutional networks

Y Yang, J Qiu, M Song, D Tao… - Proceedings of the …, 2020 - openaccess.thecvf.com
Existing knowledge distillation methods focus on convolutional neural networks (CNNs),
where the input samples like images lie in a grid domain, and have largely overlooked …

Online knowledge distillation for efficient pose estimation

Z Li, J Ye, M Song, Y Huang… - Proceedings of the IEEE …, 2021 - openaccess.thecvf.com
Existing state-of-the-art human pose estimation methods require heavy computational
resources for accurate predictions. One promising technique to obtain an accurate yet …

Cloth-changing person re-identification from a single image with gait prediction and regularization

X Jin, T He, K Zheng, Z Yin, X Shen… - Proceedings of the …, 2022 - openaccess.thecvf.com
Cloth-Changing person re-identification (CC-ReID) aims at matching the same person
across different locations over a long-duration, eg, over days, and therefore inevitably has …