Alpine pastoral systems are examples of human-managed landscapes where farming practices are traditionally adapted to semi-natural forage resources. Climate change effects will likely drive farming management with relevant socioeconomic and environmental impacts on pastoral systems and mountain communities. However, pastoral systems are complex socio-ecological systems driven by a range of drivers such as policies and institutions that trigger a range of direct and indirect feedbacks and influence the adaptation response to climate change. We present a participatory approach based on different sources and types of knowledge, building of cognitive maps with local farmers, and testing of scenarios featuring different impacts on the local pastoral system. The approach is aimed at identifying the drivers of vulnerability, understanding the cause-effect relationships between the main stress factors affecting the local pastoral system and at discerning the “concurrent” factors that enable or constrain farmers’ adaptation capacity. Our results outline that unpredictability linked to climate variability is relevant for farmers’ decisions. Nonetheless, predation is clearly the most important threat hindering the utilization of upland grasslands. The results discussed with local stakeholders point to an attentive consideration of the expected increase in farmers’ workload linked to predation, climate variability stresses, and agricultural policy. Strategies aimed at enhancing the availability of skilled shepherds result as the most effective to decrease the workload for farmers, increase the adaptive capacity of the local pastoral system, and thus contribute effectively in contrasting the abandonment of mountain grasslands.