This paper presents a novel double weight-based synthetic aperture radar (SAR) and infrared (IR) sensor fusion method (DW-SIF) for automatic ground target recognition (ATR). IR-based ATR can provide accurate recognition because of its high image resolution but it is affected by the weather conditions. On the other hand, SAR-based ATR shows a low recognition rate due to the noisy low resolution but can provide consistent performance regardless of the weather conditions. The fusion of an active sensor (SAR) and a passive sensor (IR) can lead to upgraded performance. This paper proposes a doubly weighted neural network fusion scheme at the decision level. The first weight (α) can measure the offline sensor confidence per target category based on the classification rate for an evaluation set. The second weight (β) can measure the online sensor reliability based on the score distribution for a test target image. The LeNet architecture-based deep convolution network (14 layers) is used as an individual classifier. Doubly weighted sensor scores are fused by two types of fusion schemes, such as the sum-based linear fusion scheme (αβ-sum) and neural network-based nonlinear fusion scheme (αβ-NN). The experimental results confirmed the proposed linear fusion method (αβ-sum) to have the best performance among the linear fusion schemes available (SAR-CNN, IR-CNN, α-sum, β-sum, αβ-sum, and Bayesian fusion). In addition, the proposed nonlinear fusion method (αβ-NN) showed superior target recognition performance to linear fusion on the OKTAL-SE-based synthetic database.Remote Sens. 2018, 10, 72 2 of 24 by considering both feature extractors and classifiers to cope with IR variations. The Markov tree feature [13], IR wavelet feature [14], scale invariant feature transform (SIFT) [15], histogram of oriented gradients (HOG) [16], and moment features [17,18] are recently proposed infrared features that show promising recognition results on their own applications. Simple machine learning-based classification methods, such as the nearest neighbor classifier [15], Bayesian classifier, conventional neural network, Adaboost [19], and support vector machine (SVM) [16] are used frequently to discriminate the target features for classification.SAR can measure the electromagnetic scattering property of targets under any weather and light conditions [20]. This method is used frequently to recognize a range of targets because it provides a strong radar cross section (RCS) and shape information of non-stealth targets. On the other hand, it produces many false recognitions due to speckle noise [21]. Various SAR features, such as polarimetry and transformations (log-polar, Fourier, and wavelet), are used to discriminate SAR targets [20,[22][23][24]. The standard deviation, fractal dimension, and weighted-rank fill ratio are the basic SAR features proposed in the Lincoln laboratory [22]. A genetic algorithm-based SAR feature selection method has been used to select the optimal features for target recognition [25]. Targets can be recognized by clas...