Introduction to the Special Collection on the Fragile Families Challenge

Abstract

The Fragile Families Challenge is a scientific mass collaboration designed to measure and understand the predictability of life trajectories. Participants in the Challenge created predictive models of six life outcomes using data from the Fragile Families and Child Wellbeing Study, a high-quality birth cohort study. This Special Collection includes 12 articles describing participants’ approaches to predicting these six outcomes as well as 3 articles describing methodological and procedural insights from running the Challenge. This introduction will help readers interpret the individual articles and help researchers interested in running future projects similar to the Fragile Families Challenge.

Theory In, Theory Out: The Uses of Social Theory in Machine Learning for Social Science

“…On the contrary, Lundberg et al ( 2019 ) use the statistical sampling and survey measurement theories baked into the Fragile Families Well-being Study to create the Fragile Families Challenge—a common data set computational social scientists can use to develop models predicting critical social factors like income, health, and housing. The use of theory to identify and explain important inclusion and exclusion variables have allowed research conducted during the challenge to contribute successfully to social scientific knowledge on child development (Salganik et al, 2019 ).…”

Successes and Struggles with Computational Reproducibility: Lessons from the Fragile Families Challenge

“…The benefits of computational reproducibility-and increased access to data and code, more generally-have already been articulated many times by researchers in many different fields: archaeology (Marwick, 2017), bioinformatics (Mangul et al, 2018), cell biology (Grning et al, 2018), computational fluid mechanics (Mesnard and Barba, 2017), computer systems research (Collberg and Proebsting, 2016), economics (Anderson et al, 2008;Koenker and Zeileis, 2009;Orozco et al, 2018), epidemiology (Peng et al, 2006;Coughlin, 2017;Shepherd et al, 2017), geosciences (Claerbout and Karrenbach, 1992;Gil et al, 2016;Konkol et al, 2019), high-energy physics (Chen et al, 2018), hydrology (Hutton et al, 2016), mathematical and computational biology (Schnell, 2018), machine learning (Tatman et al, 2018;Hutson, 2018), neuroscience (Crook et al, 2013;Manninen et al, 2017;Eglen et al, 2017;Mikowski et al, 2018), political science (King, 1995;Lupia and Elman, 2014;Alvarez et al, 2018), psychology (Clyburne-Sherin of the child, material hardship of the household, whether the household was evicted from their home, whether the primary caregiver participated in job training, and whether the primary caregiver lost his or her job. The choice of these outcomes was driven by our scientific goals and ethical considerations; each outcome is described in more detail elsewhere (Salganik et al, 2018;Lundberg et al, 2018). 6 For more on the construction of the Fragile Families Challenge background file, see Lundberg et al (2018).…”

“…Our process of computational reproducibility review generally involved five steps: 1) create a Docker image with the necessary software and packages; 2) move the authors' code into the Docker image and adjusting file paths to match those of the container; 3) adding the Challenge data file to the container 14 ; 4) run the authors' code inside the container in our cloud computing environment 15 ; 5) compare the results created within our computing environment to the results shown in the manuscript. We considered a manuscript to be computationally reproducible if we could run the author's code in our computing environment and regenerate 1) all the prediction.csv files (these are the files participants created to record their predictions of all six outcomes for all 4,242 families) within our error tolerance 16 ; 2) all the figures that were generated by the author's 13 The full call for papers is included in Salganik et al (2018), and here is the part most related to computational reproducibility: "What are the requirements for the open source code? The code must take the Fragile Families Challenge data files as an input and produce (1) all the figures and tables in your manuscript and supporting online materials and (2) your final predictions.…”

Friend Request Pending: A Comparative Assessment of Engineering- and Social Science–Inspired Approaches to Analyzing Complex Birth Cohort Survey Data

“…The Fragile Families Challenge (FFC) is a mass collaboration social science data challenge designed to harness the predictive power of the FFCWS data set (Salganik et al 2019). The FFC invites community members to use the data to build models that best predict six key outcomes: grade point average (GPA), grit, material hardship, eviction, job loss, and job training.…”