Understanding Data Quality through Reliability: A Comparison of Data Reliability Assessment in Three International Relations Datasets

Abstract

Although recent data creation efforts in international relations have begun to focus on issues of reliability and validity more explicitly than previously, current efforts still contain significant problems. This essay focuses on three recent data generation projects that study international relations (the ICOW, ATOP, and River Treaty datasets) and shows the successes and failures of each in assessing reliability when generating data from qualitative evidence. All three datasets attempt to generate reliable data, document the procedures used, and present indications of data reliability. However, their efforts face problems when assessing the reliability of their case selection variables, in the development of reliability indicators, and in the presentation of reliability statistics. In addition to evaluating these recent efforts to generate large‐N databases, this essay clarifies the difference between generating data from qualitative and quantitative evidence, explains the importance of reliability when coding qualitative evidence, and provides ways to improve the assessment of the quality of one’s data.

Macro-economic Factors Influencing Tourism Entrepreneurship: The Case of Kazakhstan

“…While governments have increasingly been making available various data sets in the name of good governance and transparency (Jauhari et al, 2019), the quality and reliability of such data sets can sometimes be questionable (Rothman, 2007). The data for this study are issued by the Republic of Kazakhstan government.…”

Event Data on Armed Conflict and Security: New Perspectives, Old Challenges, and Some Solutions

“…During the process of coding, or more precisely, event extraction, accuracy can be affected by oversights and inconsistency in the subjective judgments of human coders, as investigated more in detail by Laver, Benoit, and Garry (2003), Rothman (2007), and Ruggeri, Gizelis, and Dorussen (2009).…”

Military Intervention by Powerful States, 1945—2003

“…Each coder consulted at least three approved sources including scholarly studies, newspapers, chronologies of international events, and government and military records. To assess the reliability of data coded with this procedure, we assigned a random sample of 25% of the cases to two sets of student coders: a first set of student coders at the University of Georgia and a second set of student coders at Texas A&M University (Rothman, 2007). The primary political objective category (1-6) coded by the first (UGA) coder and the second (A&M) coder were identical in 23 of 29 cases.…”