Fissures and holes widely exist in rock mechanics engineering, and, at present, their failure mechanisms under complex compress and shear stress states have not been well recognized. In our work, a fracture mark, ξ, is introduced, and the kernel function of the smoothed-particle hydrodynamics (SPH) is then re-written, thus realizing the fracture modelling of the rock media. Then, the numerical models containing the fissures and holes are established, and their progressive failure processes under the compress and shear stress states are simulated, with the results showing that: (1) the improved SPH method can reflect the dynamic crack propagation processes of the rock masses, and the numerical results are in good agreement with the previous experimental results. Meanwhile, the improved SPH method can get rid of the traditional mesh re-division problems, which can be well-applied to rock failure modeling; (2) the hole shapes, fissure angles, fissure lengths, fissure numbers, and confining pressure all have great impacts on the final failure modes and peak strengths of the model; and (3) in practical engineering, the rock masses are in the 3D stress state, therefore, developing a high performance 3D SPH program and applying it to engineering in practice will be of great significance.