The DataHarmonizer: a tool for faster data harmonization, validation, aggregation and analysis of pathogen genomics contextual information

Abstract

Pathogen genomics is a critical tool for public health surveillance, infection control, outbreak investigations as well as research. In order to make use of pathogen genomics data, they must be interpreted using contextual data (metadata). Contextual data include sample metadata, laboratory methods, patient demographics, clinical outcomes and epidemiological information. However, the variability in how contextual information is captured by different authorities and how it is encoded in different databases poses challenges for data interpretation, integration and their use/re-use. The DataHarmonizer is a template-driven spreadsheet application for harmonizing, validating and transforming genomics contextual data into submission-ready formats for public or private repositories. The tool’s web browser-based JavaScript environment enables validation and its offline functionality and local installation increases data security. The DataHarmonizer was developed to address the data sharing needs that arose during the COVID-19 pandemic, and was used by members of the Canadian COVID Genomics Network (CanCOGeN) to harmonize SARS-CoV-2 contextual data for national surveillance and for public repository submission. In order to support coordination of international surveillance efforts, we have partnered with the Public Health Alliance for Genomic Epidemiology to also provide a template conforming to its SARS-CoV-2 contextual data specification for use worldwide. Templates are also being developed for One Health and foodborne pathogens. Overall, the DataHarmonizer tool improves the effectiveness and fidelity of contextual data capture as well as its subsequent usability. Harmonization of contextual information across authorities, platforms and systems globally improves interoperability and reusability of data for concerted public health and research initiatives to fight the current pandemic and future public health emergencies. While initially developed for the COVID-19 pandemic, its expansion to other data management applications and pathogens is already underway.

A General Primer for Data Harmonization

“…Meanwhile, more and more technical tools for harmonization are being made available to help data scientists, especially in the natural sciences, engage in data harmonization without having to reinvent the wheel. Some examples of tools to aid in data harmonization include: the DataHarmonizer, a standardized browser-based spreadsheet editor which is geared toward genomics data 129 and HarmonizeR, an R package which makes available an algorithm can deal with missing data in omics datasets 130 . Researchers, especially in epidemiology, may further benefit from making use Rmonize 131 , an R package which provides functions to support retrospective data harmonization, evaluation and documentation based on the guidelines developed by Fortier et al .…”

Opening Doors to Physical Sample Data Discovery, Integration, and Credit

“…3. Laboratories and data systems should provide tools that map varying, but similar, metadata requirements across different systems 61 .…”

A Practical Approach to Using the Genomic Standards Consortium MIxS Reporting Standard for Comparative Genomics and Metagenomics

“…These different implementations are supported by expert curators to validate terms and ensure compliance. Alternatively, the NMDC uses a specialized data submission tool called DataHarmonizer [30], which provides real-time validation to users and aims to lower barriers for metadata submission (Fig. 3).…”

“…Fig. 3 Screenshot of the NMDC Submission Portal (https:/ /data.microbiomedata.org/submission/home), which uses DataHarmonizer [30]. EnvO terms are shown as dropdowns for value sets along with validation checks for terms with measurement fields…”