Knowledge distillation is a technique that involves transferring knowledge from a teacher network to a student network, enabling the student to benefit from fewer parameters and computations while maintaining a comparable accuracy to the teacher. In this paper, we propose a novel distillation method that incorporates two key modules: Hard Mining Distillation and Multi-scale Feature Distillation. The Hard Mining Distillation module builds upon the vanilla knowledge distillation approach, taking inspiration from Venn diagrams. By introducing this module, we aim to enhance the students' learning by focusing on challenging areas. Moreover, we integrate the Hard Mining Distillation module into the multi-scale feature distillation process to enable more detailed learning. This method is straightforward to implement and lightweight, eliminating the complexities associated with previous studies while ensuring efficient distillation. To evaluate the effectiveness of our proposed method, we conducted quantitative experiments on the BTCV dataset. The experimental results demonstrate that our method outperforms other approaches, achieving the best performance in terms of accuracy.
Deep neural networks have demonstrated remarkable performance in medical image segmentation tasks. Nevertheless, their efficacy often comes at the cost of high model complexity and sluggish inference speeds. To address these limitations, we explore a new channel-level and pixel-level relational knowledge distillation (CPKD) method to efficiently transfer the abundant relational knowledge from the teacher network to the light-weight student network. Concretely, CPKD first captures the inter-class and intra-class semantic information of the teacher network at each channel and pixel, and then distills it to the student network in the form of knowledge. Channel-wise relational distillation utilizes the correlation between channels, while pixel-wise relational distillation utilizes the correlation between pixels. These two forms of relational knowledge complement each other in the learning process, which enables the student network to better simulate the behavior of the teacher network. Extensive experimental results on the medical image segmentation task show that CPKD can significantly improve the performance of the student network compared to other knowledge distillation methods, is highly sensitive to edge information, especially in HD95 metrics our method is far superior to the comparison methods, and has smaller model complexity at the meantime.
U-Net has become an indispensable component in medical image segmentation tasks. The characteristic of U-Net is that it produces multi-scale features, multi-scale features can provide hidden features under different views, which helps improve semantic segmentation performance. In addition, knowledge distillation, e.g., feature distillation or logit distillation, is a mechanism that can efficiently compress models. Feature distillation guides students’ feature learning by transferring feature information. In order to be able to supervise and distill these multi-scale features in feature distillation, we propose a Multi-scale Feature Distillation (MFD). MFD uses the teacher's predicted logits as the distillation target, and the students' multi-scale features of different layer as the supervision target. Nowadays, it has become a trend to decouple logits distillation. Original logits distillation can usually be divided into target classes and non-target classes. Target classes and non-target classes often play different roles in feature distillation and logits distillation. We introduce a Decoupled Multi-scale Distillation (DMD) that utilize target classes and non-target classes for feature distillation and logits distillation. When performing feature distillation, we use non-target classes for distillation, and when performing logits distillation we use target classes for distillation. Experiments on different datasets demonstrate that the DMD is effective.
Knowledge distillation requires pre-trained teachers, while self-knowledge distillation can perform knowledge distillation without pre-trained teachers. Therefore, this method is widely used in medical image classification. In this paper, we propose a reverse self-distillation Re-SKD training framework for dermatology image classification. Traditional self-knowledge distillation uses shallow networks as students and deep networks as teachers to guide teachers. The reverse self-knowledge distillation framework proposed in this article uses the shallow network as the teacher and the deep network as the student. Since the shallow layer can capture more shape and edge information in the image, and the deeper layer can learn more semantic information, we utilize features extracted from the shallow layer to construct a teacher to guide the deep teacher. We conducted experiments on the HAM10000 public data set and proved that our proposed training framework Re-SKD can significantly improve the performance of the model in skin disease classification.
Knowledge distillation is an effective model compression and performance enhancement method to instruct students by delivering soft labels or features of the instructor's logits. However, different organ classes in medical images may have similarities in shape, size, texture, etc., which may lead to mutual interference between each class. To solve this problem, we propose Class-wise restructured knowledge distillation (CWRKD). CWRKD generates class-wise features by coarse segmentation prediction of the auxiliary segmentation head with the features of the backbone, and then reconstructs it using our proposed Res module, and utilizes the instructor's features for its to bootstrap it, and also bootstrap the soft labels generated by the auxiliary segmentation head through the teacher's soft labels as a way to reduce the interference problem of similarity between each class. Experiments on different datasets show that CWRKD is effective.
Access to the requested content is limited to institutions that have purchased or subscribe to SPIE eBooks.
You are receiving this notice because your organization may not have SPIE eBooks access.*
*Shibboleth/Open Athens users─please
sign in
to access your institution's subscriptions.
To obtain this item, you may purchase the complete book in print or electronic format on
SPIE.org.
INSTITUTIONAL Select your institution to access the SPIE Digital Library.
PERSONAL Sign in with your SPIE account to access your personal subscriptions or to use specific features such as save to my library, sign up for alerts, save searches, etc.