In this paper, an SM-PB (Sparse Matrix Canonical Grid method-Physics Based Two Grid) acceleration algorithm is proposed which can be used to calculate electromagnetic scattering from two-dimensional rough surfaces with large dielectric constants. Firstly, a two-dimensional rough surface model is established based on the Monte Carlo method and Gaussian spectral function, and a conical incident wave with Gaussian characteristics is introduced to eliminate the error caused by artificial truncation of the rough surface. In the scattering calculation, the integral equation of the rough surface is processed by the SMCG algorithm, and then the matrix equation is further processed by applying the PBTG algorithm to decompose the matrix equation into the very near-field matrix, near-field matrix, and far-field matrix. The FFT method is then used to calculate the matrix vector product during the iteration for fast computation. The proposed algorithm and MoM algorithm were compared from the perspectives of computational accuracy and efficiency. Through comparison, it was found that the two algorithms produced highly consistent results, validating the effectiveness of the proposed algorithm. The proposed algorithm demonstrated a significant advantage in computational efficiency, with considerable efficiency also observed for large-scale rough surfaces. The electromagnetic scattering from rough surfaces with large dielectric constants was calculated, and the influence of the correlation distance r d and dielectric constant on the electromagnetic scattering characteristics was investigated. It was found that it is important to set a reasonable value of r d in order to balance calculation accuracy and calculation efficiency.