Understanding the long- and short-term weathering features according to rock type is very important for maintaining the original form of stone heritages. In this study, the material characteristics of a deteriorated five-story stone pagoda were compared with those of fresh rocks from its quarrying site at nearby Geumgolsan Mountain, Korea to diagnose the degree of damage and develop a comprehensive interpretation of the weathering mechanism. The stone pagoda was built from lithic tuff comprising pumice and phenocrysts such as quartz, K-feldspar, plagioclase, and mica based on plagioclase substrates; the mineralogical and geochemical characteristics are similar to those of the fresh rocks. The lithic tuff of the stone pagoda demonstrated a low ultrasonic velocity of 2863 m/s and a high water absorption rate and porosity of 9.5% and 19.2%, respectively, which are poorer than the physical properties of the fresh rocks (i.e., ultrasonic velocity of 3336 m/s, water absorption rate of 8.65%, and porosity of 17.83%). The lithological characteristics and physical properties demonstrated a considerable influence on the weathering and stability of the stone pagoda. In particular, fragments of relatively weak pumice detached from the original rock to form cavities of various sizes. These cavities introduced moisture within the rock, which produced oxides and hydroxides of iron and manganese. In addition, contaminants such as dust, salt crystals, clay minerals, and microorganisms adhering to the surface of the stone pagoda accelerated its physical, chemical, and biological weathering. The results of this study will be important for realizing the stable and long-term conservation of the five-story stone pagoda at Geumgolsan Mountain.