Knowledge distillation and student-teacher learning for visual intelligence: A review and new outlooks

L Wang, KJ Yoon - IEEE transactions on pattern analysis and …, 2021 - ieeexplore.ieee.org
Deep neural models, in recent years, have been successful in almost every field, even
solving the most complex problem statements. However, these models are huge in size with …

Towards continual reinforcement learning: A review and perspectives

K Khetarpal, M Riemer, I Rish, D Precup - Journal of Artificial Intelligence …, 2022 - jair.org
In this article, we aim to provide a literature review of different formulations and approaches
to continual reinforcement learning (RL), also known as lifelong or non-stationary RL. We …

Knowledge distillation: A survey

J Gou, B Yu, SJ Maybank, D Tao - International Journal of Computer Vision, 2021 - Springer
In recent years, deep neural networks have been successful in both industry and academia,
especially for computer vision tasks. The great success of deep learning is mainly due to its …

Meta-learning in neural networks: A survey

T Hospedales, A Antoniou, P Micaelli… - IEEE transactions on …, 2021 - ieeexplore.ieee.org
The field of meta-learning, or learning-to-learn, has seen a dramatic rise in interest in recent
years. Contrary to conventional approaches to AI where tasks are solved from scratch using …

Meta-learning with warped gradient descent

S Flennerhag, AA Rusu, R Pascanu, F Visin… - arXiv preprint arXiv …, 2019 - arxiv.org
Learning an efficient update rule from data that promotes rapid learning of new tasks from
the same distribution remains an open problem in meta-learning. Typically, previous works …

Hierarchically structured meta-learning

H Yao, Y Wei, J Huang, Z Li - International conference on …, 2019 - proceedings.mlr.press
In order to learn quickly with few samples, meta-learning utilizes prior knowledge learned
from previous tasks. However, a critical challenge in meta-learning is task uncertainty and …

Meta-learning of neural architectures for few-shot learning

T Elsken, B Staffler, JH Metzen… - Proceedings of the …, 2020 - openaccess.thecvf.com
The recent progress in neural architecture search (NAS) has allowed scaling the automated
design of neural architectures to real-world domains, such as object detection and semantic …

Empirical bayes transductive meta-learning with synthetic gradients

SX Hu, PG Moreno, Y Xiao, X Shen… - arXiv preprint arXiv …, 2020 - arxiv.org
We propose a meta-learning approach that learns from multiple tasks in a transductive
setting, by leveraging the unlabeled query set in addition to the support set to generate a …

Explaining knowledge distillation by quantifying the knowledge

X Cheng, Z Rao, Y Chen… - Proceedings of the IEEE …, 2020 - openaccess.thecvf.com
This paper presents a method to interpret the success of knowledge distillation by
quantifying and analyzing task-relevant and task-irrelevant visual concepts that are encoded …

Bootstrapped meta-learning

S Flennerhag, Y Schroecker, T Zahavy… - arXiv preprint arXiv …, 2021 - arxiv.org
Meta-learning empowers artificial intelligence to increase its efficiency by learning how to
learn. Unlocking this potential involves overcoming a challenging meta-optimisation …