Empty rituals? A qualitative study of users’ experience of monitoring & evaluation systems in HIV interventions in western India

Abstract

In global health initiatives, particularly in the context of private philanthropy and its 'business minded' approach, detailed programme data plays an increasing role in informing assessments, improvements, evaluations, and ultimately continuation or discontinuation of funds for individual programmes. The HIV/AIDS literature predominantly treats monitoring as unproblematic. However, the social science of audit and indicators emphasises the constitutive power of indicators, noting that their effects at a grassroots level are often at odds with the goals specified in policy. This paper investigates users' experiences of Monitoring and Evaluation (M&E) systems in the context of HIV interventions in western India. Six focus groups (totalling 51 participants) were held with employees of 6 different NGOs working for government or philanthropy-funded HIV interventions for sex workers in western India. Ten donor employees were interviewed. Thematic analysis was conducted. NGO employees described a major gap between what they considered their "real work" and the indicators used to monitor it. They could explain the official purposes of M&E systems in terms of programme improvement and financial accountability. More cynically, they valued M&E experience on their CVs and the rhetorical role of data in demonstrating their achievements. They believed that inappropriate and unethical means were being used to meet targets, including incentives and coercion, and criticised indicators for being misleading and inflexible. Donor employees valued the role of M&E in programme improvement, financial accountability, and professionalising NGO-donor relationships. However, they were suspicious that NGOs might be falsifying data, criticised the insensitivity of indicators, and complained that data were under-used. For its users, M& E appears an 'empty ritual', enacted because donors require it, but not put to local use. In this context, monitoring is constituted as an instrument of performance management rather than as a means of rational programme improvement.

The performativity of monitoring and evaluation in international development interventions: Building a dialogical case study of evidence-making that situates ‘the general’

“…In this way, the main task at hand acts as a kind of anchor from which analyses can focus on both ‘micro’ details, such as patterns within and between individual instances of interactions, whilst simultaneously also considering the more ‘macro’ aspects that shape differences/similarities between ‘micro’ interactions. This emphasis on exploring the macro in connection to micro- and meso-level patterns is particularly important in the study of evidence-making because of its ‘structural [international] situation’ that has already been identified in the literature (AbouZahr & MacFarlane, 2019; Cornish, 2015; Shukla et al., 2016). Namely, monitoring and evaluation processes commonly span across multiple stakeholders, often situated in different parts of the world, meaning that there are almost always imagined/non-present others, along with other Self–Other relations, and wider contexts, that situate the particular activity being observed.…”

“…Advocates of interpretivist research methods for instance emphasise how positivist quantitative evidence systems limit potentials for making inductive context-based interpretations and modifications through their focus on ascertaining ‘concrete and indisputable’ causality (Denzin, 2009; Lambert, 2016). And literature on the ‘performance management’ aspects of monitoring and evaluation identifies this quantification of practice as another form of performativity – as a technology or a mode of regulation focussed on increasing efficiency that can drive out other values such as justice or care (Ball, 2003; de St Croix, 2018; Shukla et al., 2016). Dialogical theorising on artefacts highlights how they constrain ‘mutual other-orientedness’ and reciprocity (Linell, 2001) that are essential for ethical Self–Other interdependencies.…”

“…Logframes, however, are widely recognised as problematic, owing to their linear representations of complex social practices, although an effective alternative that 'appeals to donors' has yet to be found (Hummelbruner, 2010). Qualitative and ethnographic studies of monitoring and evaluation practices in international development reveal their failure to capture local knowledge on the barriers and enablers of an intervention (Aveling, 2011;du Plessis & Lorway, 2016;Shukla, Teedon, & Cornish, 2016). Aveling (2011) suggests that donors need to create 'safe spaces' so that local partners 'feel confident to raise dilemmas and concerns' (p. 109), yet Shukla et al (2016) argue that the 'structural situation' of monitoring and evaluation, namely its use for performance management, limits the potential of these 'bottom-up processes'.…”

“…Qualitative and ethnographic studies of monitoring and evaluation practices in international development reveal their failure to capture local knowledge on the barriers and enablers of an intervention (Aveling, 2011;du Plessis & Lorway, 2016;Shukla, Teedon, & Cornish, 2016). Aveling (2011) suggests that donors need to create 'safe spaces' so that local partners 'feel confident to raise dilemmas and concerns' (p. 109), yet Shukla et al (2016) argue that the 'structural situation' of monitoring and evaluation, namely its use for performance management, limits the potential of these 'bottom-up processes'. Whilst du Plessis and Lorway (2016) identify the integration of ethnographic methods into evaluations as offering scope for facilitating better dialogues between local project staff and donors.…”

Why the pseudo matters to global health

“…Accounts which discuss power mainly in terms of it being possessed by actors in the Global North or those in higher societal strata, run the risk of overlooking the considerable agency of those in the South or in lower social echelons (Kingori & Gerrets, 2016). Similarly, saying that these actors resist power but failing to take into account the strength of the socio-economic and political structures and stratifications they come up against can over-state the effects and long-term impact of such actions (Shukla, Teedon, & Cornish, 2016).…”