Hierarchical Multi-Attention Transfer for Knowledge Distillation

Gou, Jianping; Sun, Liyuan; Yu, Baosheng; Wan, Shaohua; Tao, Dacheng

doi:10.1145/3568679

Cited by 34 publications

(7 citation statements)

References 30 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Knowledge distillation is a model-independent strategy that transfers the knowledge from the pretrained teacher network to guide the training of the student network. Knowledge distillation was originally proposed for model compression [56,57]. By learning the knowledge of the large teacher network, the lightweight student network can achieve results close to or even better than the teacher network [58][59][60].…”

Section: Knowledge Distillationmentioning

confidence: 99%

Air Traffic Flow Prediction with Spatiotemporal Knowledge Distillation Network

Shen,

Cai,

Fang

et al. 2024

Journal of Advanced Transportation

View full text Add to dashboard Cite

Accurate air traffic flow prediction assists controllers formulate control strategies in advance and alleviate air traffic congestion, which is important to flight safety. While existing works have made significant efforts in exploring the high dynamics and heterogeneous interactions of historical air traffic flow, two key challenges still remain. (1) The transfer patterns of air traffic are intricate, subject to numerous constraints and limitations such as controllers, flight regulations, and other regulatory factors. Relying solely on mining historical traffic evolution patterns makes it difficult to accurately predict the constrained air traffic flow. (2) Weather conditions exert a substantial influence on air traffic, making it exceptionally difficult to simulate the impact of external factors (such as thunderstorms) on the evolution of air traffic flow patterns. To address these two challenges, we propose a Spatiotemporal Knowledge Distillation Network (ST-KDN) for air traffic flow prediction. Firstly, recognizing the inherent future insights embedded within flight plans, we develop a “teacher-student” distillation model. This model leverages the prior knowledge of upstream-downstream migration patterns and future air traffic trends inherent in flight plans. Subsequently, to model the influence of external factors and predict air traffic flow disturbed by thunderstorm weather, we propose a student network based on the “parallel-fusion” structure. Finally, employing a feature-based knowledge distillation approach to integrate prior knowledge from flight plans and extract meteorological features, our method can accurately capture complex and constrained spatiotemporal dependencies in air traffic and explicitly model the impact of weather on air traffic flow. Experimental results on real-world flight data demonstrate that our method can achieve better prediction performance than other state-of-the-art comparison methods, and the advantages of the proposed method are particularly prominent in modeling the complicated transfer pattern of air traffic and inferring nonrecurrent flow patterns.

show abstract

Section: Knowledge Distillationmentioning

confidence: 99%

Air Traffic Flow Prediction with Spatiotemporal Knowledge Distillation Network

Shen,

Cai,

Fang

et al. 2024

Journal of Advanced Transportation

View full text Add to dashboard Cite

show abstract

“…Guo and his team [56] utilized a Recurrent Neural Network (RNN) for super-resolution data reconstruction. This enhanced the detection capabilities of UAVs and improved detection and positioning algorithms.…”

Section: Related Workmentioning

confidence: 99%

An Improved YOLOv8 to Detect Moving Objects

Safaldin,

Zaghden,

Mejdoub

2024

IEEE Access

View full text Add to dashboard Cite

Deep learning has revolutionized object detection, with YOLO (You Only Look Once) leading in real-time accuracy. However, detecting moving objects in visual streams presents distinct challenges. This paper proposes a refined YOLOv8 object detection model, emphasizing motion-specific detections in varied visual contexts. Through tailored preprocessing and architectural adjustments, we heighten the model's sensitivity to object movements. Rigorous testing against KITTI, LASIESTA, PESMOD, and MOCS benchmark datasets revealed that the modified YOLOv8 outperforms the state-of-the-art detection models, especially in environments with significant movement. Specifically, our model achieved an accuracy of 90%, a mean Average Precision (mAP) of 90%, and maintained a processing speed of 30 frames per second (FPS), with an Intersection over Union (IoU) score of 80%. This paper offers a detailed insight into object trajectories, proving invaluable in areas like security, traffic management, and film analysis where motion understanding is critical. As the importance of dynamic scene interpretation grows in artificial intelligence and computer vision, the proposed enhanced YOLOv8 detection model highlights the potential of specialized object detection and underscores the significance of our findings in the evolving field of object detection.

show abstract

“…AT [21] proposed to transfer feature attention knowledge by minimizing the L2 distance of the spatial attention maps between the teacher and student. Furthermore, several recent works [12], [13], [14], [22] also explored the channel knowledge representations and transferred them to the student. HMAT [22] proposed a hierarchical multi-attention transfer framework for knowledge distillation, which utilizes position-based, channel-based, and activation-based attention to transfer knowledge at different levels of deep representations.…”

Section: Related Work a Knowledge Distillationmentioning

confidence: 99%

“…Furthermore, several recent works [12], [13], [14], [22] also explored the channel knowledge representations and transferred them to the student. HMAT [22] proposed a hierarchical multi-attention transfer framework for knowledge distillation, which utilizes position-based, channel-based, and activation-based attention to transfer knowledge at different levels of deep representations. The knowledge can also be defined by the probability distributions of the representations.…”

Section: Related Work a Knowledge Distillationmentioning

confidence: 99%

Focal Channel Knowledge Distillation for Multi-Modality Action Recognition

Gan

Cao

et al. 2023

IEEE Access

View full text Add to dashboard Cite

The multi-modality action recognition aims to learn the complementary information from multiple modalities to improve the action recognition performance. However, there exists a significant modality channel difference, the equal transferring channel semantic features from multi-modalities to RGB will result in competition and redundancy during knowledge distillation. To address this issue, we propose a focal channel knowledge distillation strategy to transfer the key semantic correlations and distributions of multi-modality teachers into the RGB student network. The focal channel correlations provide intrinsic relationships and diversity properties of key semantics, and focal channel distributions provide salient channel activation of features. By ignoring the less-discriminative and irrelevant channels, the student can more efficiently utilize the channel capability to learn the complementary semantic features from the other modalities. Our focal channel knowledge distillation achieves 91.2%, 95.6%, 98.3% and 81.0% accuracy with 4.5%, 4.2%, 3.7% and 7.1% improvement on NTU 60 (CS), UTD-MHAD, N-UCLA and HMDB51 datasets comparing to unimodal RGB models. This focal channel knowledge distillation framework can also be integrated with the unimodal models to achieve the state-of-the-art performance. The extensive experiments show that the proposed method achieves 92.5%, 96.0%, 98.9%, and 82.3% accuracy on NTU 60 (CS), UTD-MHAD, N-UCLA, and HMDB51 datasets respectively.

show abstract

Hierarchical Multi-Attention Transfer for Knowledge Distillation

Cited by 34 publications

References 30 publications

Air Traffic Flow Prediction with Spatiotemporal Knowledge Distillation Network

Air Traffic Flow Prediction with Spatiotemporal Knowledge Distillation Network

An Improved YOLOv8 to Detect Moving Objects

Focal Channel Knowledge Distillation for Multi-Modality Action Recognition

Contact Info

Product

Resources

About