Silver fir is one of the most threatened conifer species in Croatia, especially at the western edge of its spread in Croatian Dinarides, where the decline in fir trees has resulted in significant ecological and economic issues. The aim of this study was to determine, over an 18-year monitoring period, the relationships of silver fir crown defoliation with climatic factors and structural attributes. We further analyzed the tree retention time in a given defoliation class and transition dynamics between defoliation classes, as well as the survival/mortality of trees. Data on silver fir defoliation were analyzed in two different forest types: in pure silver fir and in mixed silver fir and common beech stands. The climatic factors, primarily vegetation period air temperature, potential evapotranspiration, and dry season water deficit, were correlated with crown defoliation. Regarding the structural attributes, in the mixed stand with predominantly smaller trees, crown defoliation increased with reduced diameter at breast height, crown diameter, social class, and crown illumination. In the pure fir stand, crown defoliation increased with reduced crown diameter, greater crown asymmetry, greater crown illumination, and on trees with a stork’s nest crown. The retention time in defoliation classes differed for research sites. Transition dynamics were different only for trees in the highest defoliation class (dead trees). At the end of the study period, silver fir mortality was higher in the pure fir stand. Increased silver fir defoliation and mortality can be expected in the future, particularly in overmature stands under prolonged drought stress. Permanent forest monitoring could ensure the high-quality data needed for adaptive management of fir stands that could positively influence the structure of these stands and, thus, improve their health status.