Towards Learning Spatially Discriminative Feature Representations

Wang, Chaofei; Xiao, Jiayu; Han, Yizeng; Yang, Qisen; Song, Shiji; Huang, Gao

doi:10.1109/iccv48922.2021.00136

Cited by 20 publications

(11 citation statements)

References 26 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The current method assumes a simple fracture presence and we plan to generalize it to deal with multiple, complicated, and compound fracture presences. In addition, it will be interesting to see how human annotation can be further saved in light of [51], [52]. (c) demonstrates the network attention difference between the ResNets trained with attention guidance or without (i.e., vanilla CAM).…”

Section: Discussionmentioning

confidence: 99%

CNN Attention Guidance for Improved Orthopedics Radiographic Fracture Classification

Liao,

Shen

et al. 2022

Preprint

View full text Add to dashboard Cite

Convolutional neural networks (CNNs) have gained significant popularity in orthopedic imaging in recent years due to their ability to solve fracture classification problems. A common criticism of CNNs is their opaque learning and reasoning process, making it difficult to trust machine diagnosis and the subsequent adoption of such algorithms in clinical setting. This is especially true when the CNN is trained with limited amount of medical data, which is a common issue as curating sufficiently large amount of annotated medical imaging data is a long and costly process. While interest has been devoted to explaining CNN learnt knowledge by visualizing network attention, the utilization of the visualized attention to improve network learning has been rarely investigated. This paper explores the effectiveness of regularizing CNN network with human-provided attention guidance on where in the image the network should look for answering clues. On two orthopedics radiographic fracture classification datasets, through extensive experiments we demonstrate that explicit humanguided attention indeed can direct correct network attention and consequently significantly improve classification performance. The development code for the proposed attention guidance is publicly available on GitHub.

show abstract

Section: Discussionmentioning

confidence: 99%

CNN Attention Guidance for Improved Orthopedics Radiographic Fracture Classification

Liao,

Shen

et al. 2022

Preprint

View full text Add to dashboard Cite

show abstract

“…CCKD [31] proposed to transfer the correlation between input instances. CCM [38] proposed to match the class activation map of teacher with the class-agnostic activation map of student. In a survey of knowledge distillation [11], they discussed different forms of knowledge in three categories, including response-based [14], [26], feature-based [1], [32], [38], [43], and relation-based [31], [34], [36], [41].…”

Section: Related Work a Knowledge Distillationmentioning

confidence: 99%

“…CCM [38] proposed to match the class activation map of teacher with the class-agnostic activation map of student. In a survey of knowledge distillation [11], they discussed different forms of knowledge in three categories, including response-based [14], [26], feature-based [1], [32], [38], [43], and relation-based [31], [34], [36], [41]. However, the transferred knowledge mentioned in the existing methods always comes from pretrained teacher networks, ignoring the knowledge generated in the training process of the teacher, which we call teacher's experience.…”

Section: Related Work a Knowledge Distillationmentioning

confidence: 99%

“…A vanilla knowledge distillation [14] uses the soft targets (the outputs of the final softmax function with a temperature factor) as the teacher knowledge. Subsequently, other kinds of knowledge such as hints [32], attention maps [38], [43], probability distribution of features [30], relationships between samples [31] or layers [41] are proposed. However, they all focus on the knowledge formed after the training of the teacher, and ignore the valuable knowledge during the training process of the teacher, which is named teacher's experience.…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Learn From the Past: Experience Ensemble Knowledge Distillation

Wang¹,

Zhang²,

Song³

et al. 2022

Preprint

Self Cite

View full text Add to dashboard Cite

Traditional knowledge distillation transfers "dark knowledge" of a pre-trained teacher network to a student network, and ignores the knowledge in the training process of the teacher, which we call teacher's experience. However, in realistic educational scenarios, learning experience is often more important than learning results. In this work, we propose a novel knowledge distillation method by integrating the teacher's experience for knowledge transfer, named experience ensemble knowledge distillation (EEKD). We save a moderate number of intermediate models from the training process of the teacher model uniformly, and then integrate the knowledge of these intermediate models by ensemble technique. A self-attention module is used to adaptively assign weights to different intermediate models in the process of knowledge transfer. Three principles of constructing EEKD on the quality, weights and number of intermediate models are explored. A surprising conclusion is found that strong ensemble teachers do not necessarily produce strong students. The experimental results on CIFAR-100 and ImageNet show that EEKD outperforms the mainstream knowledge distillation methods and achieves the state-of-theart. In particular, EEKD even surpasses the standard ensemble distillation on the premise of saving training cost.

show abstract

“…Many works utilized Grad-CAM not only as a powerful tool for offline model analysis but as an embedded component in the designed deep learning model for various applications. For example, one notable capability of CAM is the target localization of a model trained only with image labels; therefore, it prevails in weakly supervised tasks, such as segmentation [4,13,16,43] and detection [38,45,50], or even knowledge distillation [39].…”

Section: Grad-based Class Activation Mapmentioning

confidence: 99%

DTFD-MIL: Double-Tier Feature Distillation Multiple Instance Learning for Histopathology Whole Slide Image Classification

Zhang¹,

Meng²,

Zhao³

et al. 2022

Preprint

View full text Add to dashboard Cite

Multiple instance learning (MIL) has been increasingly used in the classification of histopathology whole slide images (WSIs). However, MIL approaches for this specific classification problem still face unique challenges, particularly those related to small sample cohorts. In these, there are limited number of WSI slides (bags), while the resolution of a single WSI is huge, which leads to a large number of patches (instances) cropped from this slide. To address this issue, we propose to virtually enlarge the number of bags by introducing the concept of pseudo-bags, on which a double-tier MIL framework is built to effectively use the intrinsic features. Besides, we also contribute to deriving the instance probability under the framework of attentionbased MIL, and utilize the derivation to help construct and analyze the proposed framework. The proposed method outperforms other latest methods on the CAMELYON-16 by substantially large margins, and is also better in performance on the TCGA lung cancer dataset. The proposed framework is ready to be extended for wider MIL applications. The code is available at: https://github. com/hrzhang1123/DTFD-MIL

show abstract

Towards Learning Spatially Discriminative Feature Representations

Cited by 20 publications

References 26 publications

CNN Attention Guidance for Improved Orthopedics Radiographic Fracture Classification

CNN Attention Guidance for Improved Orthopedics Radiographic Fracture Classification

Learn From the Past: Experience Ensemble Knowledge Distillation

DTFD-MIL: Double-Tier Feature Distillation Multiple Instance Learning for Histopathology Whole Slide Image Classification

Contact Info

Product

Resources

About