In an open-pit mine slope, rock mass has multiple joint structures and blasting operations have an obvious influence on its stability. Therefore, accurately predicting the blasting vibration is necessary to ensure slope stability. In this study, the blasting vibration signals monitored at a blasting site with different rock masses were used to investigate the attenuation characteristics of blasting vibration through the peak particle velocity (PPV), frequency characteristics, and energy distribution of the blasting vibration signals analyzed with the time-frequency processing method. The results demonstrated that the main vibration frequency of the blasting vibration of dolomite was wider than that of shale, and these main vibration frequencies occurred at 25 kHz and 14 kHz for dolomite and shale, respectively, at a distance of 50 m from the blast area to the vibration monitoring point. With an increase in the distance from 50 m to 200 m, the main vibration frequencies decreased to less than 5 Hz. With increasing joint degree, the attenuation rate of the vibration velocity and energy attenuation of the blasting vibration increase, indicating that the structural parameters of the rock mass (such as the number of joints) have a significant impact on the attenuation law of blasting vibration. Furthermore, a modified equation that can be used for predicting PPV was developed by considering the effect of the number of joints in the rock mass on the blasting vibration. For the same ground vibration readings, the correlation factor increased from 0.8 to 0.85 for the Nicholls-USBM equation and the modified equation, respectively. The PPV of blasting under different rock masses of the Baideng open-pit phosphorite mine was used to verify the modified equation. The results show that a modified equation can be used for predicting the PPV of blasting engineering in the Baideng phosphorite mine and that the prediction accuracy is acceptable.