Improving Metadata Infrastructure for Complex Surveys: Insights from the Fragile Families Challenge

Abstract

Researchers rely on metadata systems to prepare data for analysis. As the complexity of data sets increases and the breadth of data analysis practices grow, existing metadata systems can limit the efficiency and quality of data preparation. This article describes the redesign of a metadata system supporting the Fragile Families and Child Wellbeing Study on the basis of the experiences of participants in the Fragile Families Challenge. The authors demonstrate how treating metadata as data (i.e., releasing comprehensive information about variables in a format amenable to both automated and manual processing) can make the task of data preparation less arduous and less error prone for all types of data analysis. The authors hope that their work will facilitate new applications of machine-learning methods to longitudinal surveys and inspire research on data preparation in the social sciences. The authors have open-sourced the tools they created so that others can use and improve them.

Consensus-based guidance for conducting and reporting multi-analyst studies

“…It is important that the co-analysts understand not just the available dataset but also any ancillary information that might affect their analyses (e.g., prior exclusion of outliers or handling of missing data in the blinded dataset). Providing a codebook that is accessible and understandable for researchers with different backgrounds is essential ( Kindel et al, 2019 ).…”

Consensus-based guidance for conducting and reporting multi-analyst studies

“…It is important that the co-analysts understand not just the available dataset but also any ancillary information that might affect their analyses (e.g., prior exclusion of outliers or handling of missing data in the blinded dataset). Providing a codebook that is accessible and understandable for researchers with different backgrounds is essential (25).…”

Data-Specific Functions: A Comment on Kindel et al.

“…They refer to their approach as "treating metadata as data" and suggest that the analysis of complex survey designs can be simplified if variable names are given a consistent, regular expressioncompatible nomenclature and are placed in a data set of metadata. They expect that analysts will use the data set of metadata to help them include many variables in a predictive model in a way that respects the structure of the data.Although the approach Kindel et al (2019) suggest is sound, it does not go far enough. The "metadata as data" approach builds the information from the survey's codebook-namely, response type, survey wave, and what type of respondent answered the question-into the data, but it does not encode tacit knowledge about the data set.…”

“…Although the approach Kindel et al (2019) suggest is sound, it does not go far enough. The "metadata as data" approach builds the information from the survey's codebook-namely, response type, survey wave, and what type of respondent answered the question-into the data, but it does not encode tacit knowledge about the data set.…”

“…

Elsewhere in this issue, Kindel et al (2019) describe a new approach to managing survey data in service of a prediction competition, the Fragile Families Challenge. They refer to their approach as "treating metadata as data" and suggest that the analysis of complex survey designs can be simplified if variable names are given a consistent, regular expressioncompatible nomenclature and are placed in a data set of metadata.

…”