Multilayer composite structures have significant advantages in antipenetration protection. Problems such as element distortion are more likely to occur when the FEM is used to simulate the heterogeneous composite structure against penetration with large deformation. A coupled smoothed particle hydrodynamic (SPH)-FEM is proposed to simulate the antipenetration performance of multilayer protective structures under the penetration of high-speed hemispherical-nosed projectiles. The large deformation and broken areas are calculated by the SPH method, which overcomes the problem of element distortion in FEM. In other areas, the FEM is used to improve the calculation efficiency. The results indicated that simulating multilayer plates using the coupled SPH-FEM can achieve the ballistic limit and deformation that agree with the experiment. Moreover, the deformation of single-layer, in-contact double-layer, spaced double-layer, and sandwich target plates with a core layer of water was studied and discussed. The influence of the faceplate and core layer on the penetration resistance performance was discussed in this paper by applying LSDYNA to establish the model of 3D SPH-FEM and calculate the dynamic process. In addition, the relationship between initial-residual velocity, deformation, and damage failure behavior was obtained.