Monitoring Open Science as Transformative Change: Towards a Systemic Framework

Abstract

Background: Following a flurry of policies for Open Science (OS), there is now a wave of initiatives to monitor its adoption. However, the great diversity of understandings and activities associated related to Open Science makes monitoring very challenging. There is a danger that by focusing on what can be readily observed (e.g. publications) many other OS activities are overlooked (e.g. participation), with a potential narrowing of OS scope, ‘street-light’ effects, and deviation from the values of OS.Methods:We have conducted desk research to analyse existing OS conceptualisations and monitoring efforts against the evaluation literature. Since Open Science can be understood as a systemic transformation of the research system, we have borrowed in particular concepts from Transformative Innovation Policies (TIPs) frameworks which aim at evaluating socio-technical transitions.Results: In accordance with the notion of OS as a systemic transformation, we propose that OS should imply a change in monitoring frameworks for OS. Just like the shift from the ‘linear model’ to ‘innovation systems’ models in the 1980s was accompanied with new statistical concepts and tools (e.g. the OECD’s Oslo manual and innovation surveys), tracking OS adoption requires a far-reaching change in monitoring. We propose that the new monitoring should shift towards: (i) a systemic perspective which considers the various actions related to OS, including policies and outputs (e.g. datasets) but also processes (e.g. participatory events), outcomes (e.g. use of knowledge in policy) and expected impacts (e.g. increased attention to societal problems); (ii) implementation of monitoring as learning (rather than accountability or benchmarking) for strategic perspectives and reflexive evaluation; (ii) mapping the directionality of the activities and the values associated with the choices in directions. Conclusions:A monitoring framework for OS requires a profound change in monitoring framework and practices. The scope should broaden from outputs (such as publications) towards the processes of connection that make science ‘open’ (usage, co-creation and dialogue), as well as towards outcomes (changes in practices) and the longer-term impacts that reflect the values and normative commitments of OS.