Strengthening Robustness Under Adversarial Attacks Using Brain Visual Codes

Abstract

The vulnerability of computational models to adversarial examples highlights the differences in the ways humans and machines process visual information. Motivated by human perception invariance in object recognition, we aim to incorporate human brain representations for training a neural network. We propose a multi-modal approach that integrates visual input and the corresponding encoded brain signals to improve the adversarial robustness of the model. We investigate the effects of visual attacks of various strengths on an image classification task. Our experiments show that the proposed multi-modal framework achieves more robust performance against the increasing amount of adversarial perturbation than the baseline methods. Remarkably, in a black-box setting, our framework achieves a performance improvement of at least 7.54% and 5.97% on the MNIST and CIFAR-10 datasets, respectively. Finally, we conduct an ablation study to justify the necessity and significance of incorporating visual brain representations.cortex. The image-fMRI dataset comprises N paired samplesThe goal of a multi-modal 313 classifier is to learn a mapping function f : X , V → Y . 314 In terms of architecture, image classifier is based on 315 ResNet18 architecture [64]. fMRI classifier adopts a 2-layer 316 deep neural network (DNN) with two fully connected lay-317 ers separated by a non-linear activation function. While the 318 convolutional network networks are considered effective for 319 extracting information from images, we find simple feed-320 forward DNNs sufficient for performing fMRI classification. 321 The visual information from the image and brain codes 322 from the fMRI is integrated by extracting the visual feature 323 vector f x from the image and fMRI feature vector f v . Specif-324 ically, these extracted features f x and f v correspond to penul-325 timate or pre-classification layer activations for image and 326 fMRI classifiers. Then, the multi-modal method integrates 327 the visual and fMRI feature vectors into a single vector via 328 concatenation f xv = f x f v . Finally, using the cross-entropy 329 loss function, a fully-connected layer with a softmax predicts 330 the output class probability from f xv . 331 D. IMPLEMENTATION DETAILS 332 We train our networks using NVIDIA TITAN RTX GPUs. 333 The encoder and decoder are trained with the Adam opti-334 mization for 100 epochs with the learning rate of 1e-4 and 335 1e-3, and batch size 32 and 50, respectively. The main multi-336 modal framework was trained for 50 epochs using learning 337 rate 1e-2 and batch size 64. We ran the PGD and MIM attacks 338 for 10 steps.