Climate change is impacting the sustainability of food systems through shifts in natural and human dimensions of agroecosystems that influence farmer livelihoods, consumer choices, and food security. This paper highlights the need for climate studies on specialty crops to focus not only on yields, but also on quality, as well as the ability of agroecological management to buffer climate effects on quality parameters. Crop quality refers to phytonutrient and secondary metabolite profiles and associated health and sensory properties that influence consumer buying decisions. Through two literature reviews, we provide examples of specialty crops that are vulnerable to climate effects on quality and examples of climate-resilient agroecological strategies. A range of specialty crops including fruits, vegetables, tree nuts, stimulants, and herbs were identified to respond to climate variables with changes in quality. The review on climate-resilient strategies to mitigate effects on crop quality highlighted a major gap in the literature. However, agricultural diversification emerged as a promising strategy for climate resilience more broadly and highlights the need for future research to assess the potential of diversified agroecosystems to buffer climate effects on crop quality. We integrate the concepts from our literature review within a socio-ecological systems framework that takes into account feedbacks between crop quality, consumer responses, and agroecosystem management. The presented framework is especially useful for two themes in agricultural development and marketing, nutrition-sensitive agriculture and terroir, for informing the design of climate-change resilient specialty crop systems focused on management of quality and other ecosystem services towards promoting environmental and human wellbeing.