Inspired by neural networks based on traditional electronic circuits, optical neural networks (ONNs) show great potential in terms of computing speed and power consumption. Though some progress has been made in devices and schemes, ONNs are still a long way from replacing electronic neural networks in terms of generalizability. Here, we present a complex optical neural network (cONN) for holographic image recognition, within which a high-speed parallel operating unit for complex matrices is proposed, targeting the real-imaginary-splitting and column splitting. Based on the proposed cONN, we have numerically demonstrated the training-recognition process on our cONN for holographic images converted from handwritten digit datasets, achieving an accuracy of 90% based on the back-propagation algorithm. Our training-verification integrated architecture will enrich the further development and applications of on-chip photonic matrix computing.