Holographic memory offers high-capacity optical storage with rapid data readout and long-term durability. Recently, read data pages have been classified using digital deep neural networks (DNNs). This approach is highly accurate, but the prediction time hinders the data readout throughput. This study presents a diffractive DNN (D2NN)-based classifier for holographic memory. D2NNs have so far attracted a great deal of attention for object identification and image transformation at the speed of light. A D2NN, consisting of trainable diffractive layers and devoid of electronic devices, facilitates high-speed data readout. Furthermore, we numerically investigated the classification performance of a D2NN-based classifier. The classification accuracy of the D2NN was 99.7% on 4-bit symbols, exceeding that of the hard decision method.