Polycrystalline materials are extensively employed in industry. Its surface roughness has a great impact on the working performance. The material defect, particularly grain boundary, has great impact on the achieved surface roughness of polycrystalline materials. However, it is difficult to establish a purely theoretical model for surface roughness with consideration of the grain boundary effect using conventional analytical methods. In this work, a theoretical and deep learning hybrid model for predicting the surface roughness of diamond-turned polycrystalline materials is proposed. The kinematic-dynamic roughness component in relation to the tool profile duplication effect, work material plastic side flow, relative vibration between diamond tool and workpiece, etc. is theoretically calculated. The material-defect roughness component is modelled with a cascade forward neural network. In the neural network, the relative tool sharpness, work material properties (misorientation angle and grain size), and spindle rotation speed are configured as input variables. The material-defect roughness component is set as the output variable. To validate the developed model, polycrystalline copper with a gradient distribution of grains prepared by friction stir processing is machined with various processing parameters and different diamond tools. Compared with the previously developed model, obvious improvement on the prediction accuracy is observed with this hybrid prediction model. Based on this model, the influences of different factors on the surface roughness of polycrystalline materials are discussed. The influencing mechanism of the misorientation angle and grain size is quantitatively analysed. Two fracture mode, including transcrystalline and intercrystalline fracture at different relative tool sharpness is observed. Meanwhile, optimal processing parameters are obtained with a simulated annealing algorithm. Cutting experiments are performed with the optimal parameters, and a flat surface finish with Sa 1.314 nm is finally achieved.