Online cross-layer knowledge distillation on graph neural networks with deep supervision

Guo, Jingjing; Chen, Defang; Wang, Can

doi:10.1007/s00521-023-08900-7

Cited by 3 publications

(1 citation statement)

References 34 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…By adopting this strategy, we optimize efficiency, allowing particles to identify the optimal global position without imposing excessive computational demands 59 . Traditional compression methods for neural network models often raise concerns about their potential negative impact on model robustness 3 , 14 , 15 , 17 , 19 . In our approach, we avoid directly reducing or compressing the model, as such actions have been associated with detrimental effects on model parameters.…”

Section: Related Workmentioning

confidence: 99%

Defense against adversarial attacks: robust and efficient compressed optimized neural networks

Kraidia,

Ghenai,

Belhaouari

2024

Sci Rep

View full text Add to dashboard Cite

In the ongoing battle against adversarial attacks, adopting a suitable strategy to enhance model efficiency, bolster resistance to adversarial threats, and ensure practical deployment is crucial. To achieve this goal, a novel four-component methodology is introduced. First, introducing a pioneering batch-cumulative approach, the exponential particle swarm optimization (ExPSO) algorithm was developed for meticulous parameter fine-tuning within each batch. A cumulative updating loss function was employed for overall optimization, demonstrating remarkable superiority over traditional optimization techniques. Second, weight compression is applied to streamline the deep neural network (DNN) parameters, boosting the storage efficiency and accelerating inference. It also introduces complexity to deter potential attackers, enhancing model accuracy in adversarial settings. This study compresses the generative pre-trained transformer (GPT) by 65%, saving time and memory without causing performance loss. Compared to state-of-the-art methods, the proposed method achieves the lowest perplexity (14.28), the highest accuracy (93.72%), and an 8 × speedup in the central processing unit. The integration of the preceding two components involves the simultaneous training of multiple versions of the compressed GPT. This training occurs across various compression rates and different segments of a dataset and is ultimately associated with a novel multi-expert architecture. This enhancement significantly fortifies the model's resistance to adversarial attacks by introducing complexity into attackers' attempts to anticipate the model's prediction integration process. Consequently, this leads to a remarkable average performance improvement of 25% across 14 different attack scenarios and various datasets, surpassing the capabilities of current state-of-the-art methods.

show abstract

Section: Related Workmentioning

confidence: 99%