Reproducibility and Replicability in Geographical Analysis

Abstract

The scientific method is predicated on the assumption that research designs and results can be reproduced and replicated. However, recent findings in some disciplines suggest that many studies fail to reach this standard, moving issues surrounding reproducibility and replicability forward into the research agenda of those fields. While the topic has yet to become a point of controversy in geography, the intricacies of geographic phenomena and spatial data analysis make the field vulnerable to criticism. This commentary discusses how uncertainties related to the conception, measurement, analysis, and communication of geographic analyses contribute to difficulties in the reproduction and replication of geographic research. Investigating how these uncertainties collectively impact the reproducibility and replicability of spatial data analyses should be a critical focus of future Geographical Analysis research. A call to action for geographers to improve the reproducibility and replicability of their work and specific recommendations on how Geographical Analysis might facilitate this process conclude the commentary.

Opening practice: supporting reproducibility and critical spatial data science

“…If we are to consider the stages in the development of software tools following those listed in Table 1, perhaps an interesting further development is that stage 1 has been revisited, particularly in terms of ideas of reproducibility. This captures general view that data analysis carried out in research should, wherever reasonable, be capable of reproduction by a third party (Singleton et al 2016;Wang 2016;Shannon and Walker 2018;Nüst et al 2018;Singleton and Arribas-Bel 2019;Kedron et al 2019). In essence, this requires that any reporting of data analysis should provide the computational procedure used and the data, in addition to a presentation of the outcome of the analysis.…”

“…These have emerged for a number of reasons: software costs, distrust of the 'black box' where data are processed without disclosure of the processing method, recognition of the scientific advantages of working in an open source environment where new methods are typically available several years before their availability in commercial software, as well as the wider practice of user community-generated software extensions and improvements-social (geo-) computation, and reproducibility. The distrust of the black box reflects a widely held view that data analysis and research should, wherever reasonable, be capable of reproduction/replication by a third party, where reproducibility is defined is the exact duplication of the results using the same materials, and replicability means confirming original conclusions (Nüst et al 2018), for example, with new data (Kedron et al 2019), both as a scientific credo and to avoid further 'climategates' (Campbell 2010), in which a key issue was that the researchers involved would not release their data (or their code). The need for such transparency derives from the dangers of uncritical acceptance of black box spatial analyses, and the potential for erroneous results of such approaches precisely because 'there is less of a requirement to think about the underlying processes that are being implemented' (Singleton et al 2016(Singleton et al , p. 1512 In essence, a reproducible research philosophy is one which allows all aspects of the answer generated by any given analysis to be tested.…”

Quantitative Geography III: Future Challenges & Challenging Futures

“…Whether this transfers to geography remains to be seen. Brunsdon (2016) outlines the challenges posed for reproducibility in geography, and Kedron et al (2020) provides a further elaboration the theoretical challenges to replication in geography. Further, Retraction Watch shows retractions in a few geography journals, but these are rare and will hopefully remain so.…”

Urban Street Network Analysis in a Computational Notebook

“…Urban planning and its related disciplines benefit accordingly from the growing adoption of computational notebooks in pedagogy, research, and practice. Computation is increasingly central to the field and its practitioners benefit from open and reproducible approaches to analyzing urban data and predicting city futures (Kedron et al 2019;Kontokosta 2018;Batty 2019). In the Python universe, for example, numerous new tools now exist to support urban analytics and planning processes, including data wrangling/analysis (pandas), visualization (matplotlib), geospatial wrangling/analysis (geopandas), spatial data science and econometrics (pySAL), mapping (cartopy), web mapping (folium), network analysis (NetworkX), land use modeling/simulation (Ur-banSim), activity-based travel modeling (ActivitySim), and computational notebooks themselves (Jupyter).…”