To study the effect of material properties on the surface morphology of potassium dihydrogen phosphate (KDP) crystals, an ultra-precision fly cutting machine tool with a single-point diamond tool was used to perform a cutting experiment on (100) crystal plane of the KDP crystal. The elastic modulus, shear modulus, hardness, and dislocation of KDP crystals are taken into the cutting force model by introducing the strain gradient plasticity theory. Since the size effect and dynamic response will affect the surface roughness during ultra-precision machining, the surface roughness of workpieces in ultra-precision fly cutting is hard to predict. Based on the previously established strain gradient plasticity theoretical model, cutting force model, and the dynamic characteristics of the ultra-precision fly cutting system, a surface morphology prediction model under the influence of KDP crystal material properties was established. Finally, the accuracy of the surface morphology prediction model was verified by ultra-precision fly cutting experiments, and identified the frequency range of the characteristic signal caused by the anisotropy of the KDP crystal from the frequency, thereby verifying the KDP crystal material properties has a significant effect on the surface of the machined workpiece roughness.