The mountainous areas of the Loess Plateau in China are crisscrossed with ravines, fragile ecological environment, and frequent landslide hazards. Landslides often cause building collapses and casualties, seriously affecting the sustainability of economic and social development in the region. In order to study the damage of loess landslides to village and town buildings, the paper takes the Xingwang landslide in Fugu County, Shaanxi Province, China as an example, and uses discrete element method and finite element method to study the dynamic process of landslide and its damage to houses. Firstly, the geological characteristics of the landslide were identified by means of investigation, surveying, engineering exploration and geotechnical testing. Secondly, a three-dimensional numerical model of the landslide area was established by using the particle flow code system (PFC3D). Finally, the entire movement process of the landslide was simulated, and the impact damage to houses induced from landslide was analyzed and predicted. The results show that the whole movement of the landslide lasted a total of 180 s, with a maximum average velocity of 2.01 m/s and a maximum average displacement of 73.7 m. The first and second rows of houses located at the foot of the landslide will suffer serious damage, with most bricks displacement ranging from 0.1 to 2.5 m and a maximum displacement of 10.3 m, posing a serious safety risk to the houses. Only a portion of the third row houses will be damaged, and the fourth row houses will not be threatened by landslide. By comparing with the prediction results of other methods and the current situation of buildings deformation, the results of this paper have a certain credibility. This study provides a numerical method for quantitative assessment of the risk and building damage for loess landslide, which can be used as a reference. It also provides technical support for formulating hazard prevention and reduction plans for the Xingwang landslide.