A semi-automated approach to policy-relevant evidence synthesis: Combining natural language processing, causal mapping, and graph analytics for public policy

Abstract

Although causal evidence synthesis is critical for the policy sciences – whether it be analysis for policy or analysis of policy – its repeatable, systematic, and transparent execution remains challenging due to the growing volume, variety, and velocity of policy-relevant evidence generation as well as the complex web of relationships within which policies are usually situated. To address these shortcomings, we developed a novel, semi-automated approach to synthesizing causal evidence from policy-relevant documents. Specifically, we propose the use of natural language processing (NLP) for the extraction of causal evidence and subsequent homogenization or normalization of the varied text, causal mapping for the collation, visualization, and summarization of complex interdependencies within the policy system, and graph analytics for further investigation of the structure and dynamics of the causal map. We illustrate this approach by applying it to a collection of 28 articles on the emissions trading scheme (ETS), a policy instrument of increasing importance for climate change mitigation. In all, we find 300 variables and 284 cause-effect pairs in our input dataset (consisting of 4524 sentences), which are reduced to 70 unique variables and 119 cause-effect pairs after normalization. We create a causal map depicting these and analyze it subsequently to obtain systemic perspective as well as policy-relevant insight on the ETS that is broadly consistent with select manually conducted, previous meta-reviews of the policy instrument. We conclude that, despite its present limitations, this approach can help synthesize causal evidence for policy analysis, policymaking, and policy research.