Compact computational in‐line holography based on deep learning is an attractive single‐shot approach to image microparticles dispersed in 3D volume. The particle shape contains valuable information for species classification, but the dataset acquisition process for network training suffers from labor consuming and low efficiency due to the complex and varied 2D shapes of different particle species. Herein, a moment‐based shape‐learning holography (MSLH) is proposed where the shape of a microparticle is mathematically characterized using varying weights of Zernike moments. By decomposing and recombining feature shapes of pollen particles into numerous new characteristic shapes as incremental data, MSLH improves the efficiency of dataset preparation. The depth‐encoded shape‐learning is achieved using a U‐net with self‐attention mechanism, which enables fast axial depth determination. The shape reconstruction uses a wavelet‐based method with more explicit physical meanings, making MSLH a hybrid data‐and‐model driven approach that requires fewer primary data. Validation results show that MSLH achieves high accuracy in axial position and shape reconstruction, while maintaining good classification effectiveness. It is believed that MSLH is an easy‐to‐setup, efficient‐to‐construct, and fast‐to‐output approach for shape‐based classifications of 3D distributed microparticles in dynamic fluid.