Measuring the Hard-to-Measure in Development: Dimensions, Measurement Challenges, and Responses

Abstract

As set out in the 2030 Agenda, sustainable development is universal, integrated, and indivisible, and balances economic, social, and environmental dimensions. Initiatives that aim to improve development attempt to address entrenched issues, where previous efforts have been insufficient or adequate responses are not known. They operate under conditions of uncertainty and complexity. Increasingly they involve multi-component or package interventions, delivered by and relating to diverse sets of stakeholders pursuing different, sometimes competing interests, and working in shifting contexts. Each of these factors carry implications for measurement and pose distinct threats to validity and reliability. In recent years, there has been a greater acknowledgement of these challenges and the need for robust mixed methods and more flexible, adaptive approaches. At the same time, how best to gather, combine, and interpret multiple sources of evidence covering multi-dimensional aspects of economic and social development, and communicate and use this evidence to guide policy and programming remains a formidable challenge. As applied researchers attempting to gather credible evidence on the effects and trajectories of efforts to improve development, we must continually ask not only "Are we measuring it right?", but also, more fundamentally, "Are we measuring the right thing?" Based on common challenges that arose through development initiatives in a variety of contexts, this contribution explores four hard-to-measure dimensions of development. In particular, we discuss abstract, multi-dimensional concepts, processes, and issues; challenging settings where there are unpredictable, sudden, or frequent shifts in the environment; multiple, uncertain pathways of change; and multi-layer implementing structures such as cross-sector partnerships or regional/

Beyond Indicators and Success Stories: An Emerging Method to Assess Social Learning in Large-Scale Transdisciplinary Research Programs

“…Effectively documenting SL processes to assess their influences on large-scale programs presents challenges on several fronts. Buffardi et al (2019) point to four "hard to measure" dimensions of assessing change in large-scale programs, all of which feature, to varying extents, within evidence on programming for climate change and sustainable development:…”

“…Yet, the development of methods for assessing SL processes and their outcomes within these programs remains slow (Ensor and Harvey, 2015;van Epp and Garside, 2019). This is, in part, due to the complexity of both large multi-partner programs, and of learning and social change (Buffardi et al, 2019). However, failure to effectively assess SL and its contributions to wider programmatic outcomes can limit our ability and inclination to invest in the strengths of these approaches, and our understanding of what their limits might be.…”