Earth-based building materials are increasingly valued in green design for their low embodied energy, humidity-buffering ability, and thermal stability. These materials perform well in warm dry climates, but greater understanding of long-term durability is needed for successful adoption in colder and/or wetter climates. The presence of stabilizers dramatically improves resistance to surface erosion from wind and rain, compared to unstabilized adobe and cob counterparts, and the influences of soil composition, fiber type, and diverse binders, on rain and wind surface erosion have been investigated in detail. Frost and freeze-thaw resistance, however, have been less well-studied, despite strong interest in stabilized earth materials in northern North America, Europe, and Asia. In particular, recent studies have relied on a widespread misunderstanding of the mechanism by which frost damage occurs in porous materials that will impede efforts to create valid models for material design and improvement. In addition, the influence of radiative thermal stresses on wall surfaces has been overlooked in favor of focus on ambient air temperatures. Here, we apply contemporary understanding of cracking by segregated ice growth to develop a macroscopic damage index that enables comparison between performance of different materials subject to different weather patterns. An examination of predicted damage patterns for two stabilized earth building materials and two conventional materials in twelve cities over two time periods reveals the dominant factors that govern frost vulnerability. We find that the frost resilience of earth building materials is comparable to that of the conventional materials we examined, and that assessments that neglect expected variations in water content by assuming full saturation are likely to yield misleading results. Over recent years, increased winter temperatures in several cities we examined predict reduced material vulnerability to frost damage, but we also find that accompanying increases in humidity levels have made some cities much more vulnerable.