A molecular dynamics model of the diamond abrasive polishing the single crystal silicon is established. Crystal surfaces of the single crystal silicon in the Y-direction are (010), (011), and (111) surfaces, respectively. The effects of crystallographic orientations on polishing the non-continuous single crystal silicon surfaces are discussed from the aspects of surface morphology, displacement, polishing force, and phase transformation. The simulation results show that the Si(010) surface accumulates chips more easily than Si(011) and Si(111) surfaces. Si(010) and Si(011) workpieces are deformed in the entire pore walls on the entry areas of pores, while the Si(111) workpiece is a local large deformation on entry areas of the pores. Comparing the recovery value of the displacement in different workpieces, it can be seen that the elastic deformation of the A side in the Si(011) workpiece is larger than that of the A side in other workpieces. Pores cause the tangential force and normal force to fluctuate. The fluctuation range of the tangential force is small, and the fluctuation range of the normal force is large. Crystallographic orientations mainly affect the position where the tangential force reaches the maximum and minimum values and the magnitude of the decrease in the tangential force near the pores. The position of the normal force reaching the maximum and minimum values near the pores is basically the same, and different crystallographic orientations have no obvious effect on the drop of the normal force, except for a slight fluctuation in the value. The high-pressure phase transformation is the main way to change the crystal structure. The Si(111) surface is the cleavage surface of single crystal silicon, and the total number of main phase transformation atoms on the Si(111) surface is the largest among the three types of workpieces. In addition, the phase transformation in Si(010) and Si(011) workpieces extends to the bottom of pores, and the Si(111) workpiece does not extend to the bottom of pores.