Date fruit is among the major crops in the middle-east region, where millions of tons are harvested every year. Date is a healthy fruit, which involves sugars, minerals and vitamins. In addition, it helps preventing human body from several diseases such as cancer and heart diseases. Date sorting is a fundamental step in the date industry. However, manually conducting such an operation, by human labors, is expensive and time-consuming. In this paper, we propose a method for classifying the type of date fruit by incorporating supervised and unsupervised deep networks. Specifically, we use discriminant correlation analysis (DCA) algorithm to fuse features learned from convolution neural networks (VGG-F) and an unsupervised network called PCANet. DCA jointly performs feature fusion and dimensionality reduction with a low computational complexity. To carry out experiments, we introduce a new benchmark dataset of date fruit images from 20 date varieties. Our benchmark is, to the best of our knowledge, the largest one in terms of number of varieties. Note that the dataset is publicly available at https://unsat.000webhostapp.com/dataset . Experimental results demonstrate the utility of DCA as well as the complementarity of the fused features. It has also been shown the effectiveness of the proposed method compared to several relevant methods.