A ray tracing method based on the geometrical optics and physical optics approximation is proposed for calculation of the scattering from rough metal surface with rough coating in terahertz regime. Geometrical optics is utilized to describe the wave propagation concerning the reflection and transmission, whereas physical optics is applied to obtain the scattering fields by equivalent currents on the surfaces. The method provides a reliable treatment of scattering from rough layers involving identical/different roughness. The scattering predictions of the rough layers are compared to the multilevel fast multipole algorithm (MLFMA) to demonstrate the efficiency and accuracy of the proposed method. Once validated, we use the method to carry out a sensitivity study of the bistatic radar scattering cross section to variations in layer parameters of interest in terahertz coating target detection, such as the roughness of the coating and metal surface, thickness of the coating layer. INDEX TERMS Electromagnetic scattering, terahertz regime, ray tracing, rough coating.