Stage-by-stage knowledge distillation

Mengya Gao; Liang Wan; Yujun Wang; Jinyan Chen

doi:10.1117/12.2589331

27 January 2021 Stage-by-stage knowledge distillation

Mengya Gao, Liang Wan, Yujun Wang, Jinyan Chen

Proceedings Volume 11720, Twelfth International Conference on Graphics and Image Processing (ICGIP 2020); 117202M (2021) https://doi.org/10.1117/12.2589331
Event: Twelfth International Conference on Graphics and Image Processing, 2020, Xi'an, China

Abstract

Knowledge Distillation (KD) aims at using a low-capacity model, called student, to learn from a high-capacity one, termed as teacher, such that the performance of student can be improved. Previous KD methods typically train a student by minimizing a task-related loss and the KD loss simultaneously, with the help of a loss weight hyper-parameter to balance these two terms. In this work, we propose to first transfer the backbone knowledge from a teacher to the student, and then only learn the task-head of the student network. Such a training decomposition alleviate the use of loss weight, which can be hard to define. This allows our method to be easily applied to different datasets or tasks with strong stability. Importantly, the decomposition permits the core of our method, Stage-by-Stage Knowledge Distillation (SSKD), which facilitates progressive feature mimicking from teacher to student. Extensive experiments on CIFAR-100 and ImageNet suggest that SSKD significantly narrows down the performance gap between student and teacher, outperforming state-of-the-art approaches. We also demonstrate the generalization ability of SSKD on object detection on COCO dataset. On both tasks SSKD shows significant improvements.

Citation Download Citation

Mengya Gao, Liang Wan, Yujun Wang, and Jinyan Chen "Stage-by-stage knowledge distillation", Proc. SPIE 11720, Twelfth International Conference on Graphics and Image Processing (ICGIP 2020), 117202M (27 January 2021); https://doi.org/10.1117/12.2589331

ACCESS THE FULL ARTICLE

INSTITUTIONAL
Select your institution to access the SPIE Digital Library.

SELECT YOUR INSTITUTION

PERSONAL
Sign in with your SPIE account to access your personal subscriptions or to use specific features such as save to my library, sign up for alerts, save searches, etc.

PERSONAL SIGN IN

No SPIE Account? Create one

PURCHASE THIS CONTENT

SUBSCRIBE TO DIGITAL LIBRARY

50 downloads per 1-year subscription

Members: $195

Non-members: $335 ADD TO CART

25 downloads per 1 - year subscription

Members: $145

Non-members: $250 ADD TO CART

PURCHASE SINGLE ARTICLE

Includes PDF, HTML & Video, when available