The study of dynamic soil‐structure interaction is significant to civil engineering applications, such as machine foundation vibration, traffic‐induced vibration, and seismic dynamic response. The scaled boundary finite element method (SBFEM) is a semi‐analytical algorithm, which is used to solve the dynamic response of a three‐dimensional infinite soil. It can automatically satisfy the radiation boundary condition at infinity. Based on the dynamic stiffness matrix equation obtained by the modified SBFEM, a continued fraction algorithm is proposed to solve the dynamic stiffness matrix of layered soil in the frequency‐domain. Then, the SBFEM was coupled with the finite element method (FEM) at the interface to solve the dynamic stiffness matrices of the rigid surface/buried foundation. Finally, the mixed‐variable algorithm was used to solve the three‐dimensional transient dynamic response of the foundation in the time domain. Numerical examples were performed to verify the accuracy of the proposed algorithm in solving the dynamic stiffness matrix of the infinite domain in the frequency domain and the dynamic transient displacement response of the foundation in the time domain. Compared with the previous numerical integration technique, the dynamic stiffness matrix in the frequency domain calculated by using the proposed algorithm has higher accuracy and higher efficiency.