Pairs and Pairix: a file format and a tool for efficient storage and retrieval for Hi-C read pairs

Lee, Mi Kyung; Bakker, Clara; Vitzthum, Carl; Alver, B.; Park, Peter J.

doi:10.1093/bioinformatics/btab870

“…The ENCODE Hi-C pipeline has been developed with the Aiden lab using their Juicer suite of software tools 38 , with some updates to mapping parameters and chimeric read handling. There are essentially five steps in the pipeline (Fig 8A ); mapping (with bwa-mem) and filtering plus Pairix 39 to form a set of contacts, or pairs file. The genome is then binned into 14 resolutions (between 10bp and 2.5Mbp) by Juicer to form contact matrix (.hic) files.…”

Section: The Encode Hi-c Pipelinementioning

confidence: 99%

The ENCODE Uniform Analysis Pipelines

Hitz

¹

,

Lee

²

,

Jolanki

³

et al. 2023

Preprint

View full text Add to dashboard Cite

The Encyclopedia of DNA elements (ENCODE) project is a collaborative effort to create a comprehensive catalog of functional elements in the human genome. The current database comprises more than 19000 functional genomics experiments across more than 1000 cell lines and tissues using a wide array of experimental techniques to study the chromatin structure, regulatory and transcriptional landscape of the Homo sapiens and Mus musculus genomes. All experimental data, metadata, and associated computational analyses created by the ENCODE consortium are submitted to the Data Coordination Center (DCC) for validation, tracking, storage, and distribution to community resources and the scientific community. The ENCODE project has engineered and distributed uniform processing pipelines in order to promote data provenance and reproducibility as well as allow interoperability between genomic resources and other consortia. All data files, reference genome versions, software versions, and parameters used by the pipelines are captured and available via the ENCODE Portal. The pipeline code, developed using Docker and Workflow Description Language (WDL; https://openwdl.org/) is publicly available in GitHub, with images available on Dockerhub (https://hub.docker.com), enabling access to a diverse range of biomedical researchers. ENCODE pipelines maintained and used by the DCC can be installed to run on personal computers, local HPC clusters, or in cloud computing environments via Cromwell. Access to the pipelines and data via the cloud allows small labs the ability to use the data or software without access to institutional compute clusters. Standardization of the computational methodologies for analysis and quality control leads to comparable results from different ENCODE collections - a prerequisite for successful integrative analyses.

show abstract

“…The modular structure of pairtools and its usage of the .pairs format 28 already make it useful in many pipelines. pairtools is used in the 4DN pipeline (standard Hi-C) 3 , the PORE-C pipeline (multi-way Hi-C) 75 , HI-CAR nf-core pipeline (open-chromatin-associated contacts) 76 , and iMARGI pipelines (RNA-DNA contacts) 49 77 .…”

Section: Discussionmentioning

confidence: 99%

“…Parse detects such cases, and merges the alignments, “rescuing” the true contact pair (Supplementary Figure 1d). The output of parse adheres to the standard format .pairs 28 , discussed below.…”

Section: Essential Building Blocks For 3c+ Pair Processingmentioning

confidence: 99%

“…Both parse and Parse2 output contact tables in a text tab-separated format called .pairs, designed by the NIH 4DN consortium 28 . As a text format, .pairs has several advantages over custom binary formats: (i) text tables can be processed in all programming languages, (ii) are easily piped between individual CLI tools, and (iii) have a set of highly efficient utilities for sorting (Unix sort ), compression ( bgzip/lz4 ), and random access ( tabix 41 /pairix 28 ).…”

Section: Essential Building Blocks For 3c+ Pair Processingmentioning

confidence: 99%

See 1 more Smart Citation

Pairtools: from sequencing data to chromosome contacts

Abdennur

¹

,

Fudenberg

²

,

Flyamer

³

et al. 2023

Preprint

View full text Add to dashboard Cite

The field of 3D genome organization produces large amounts of sequencing data from Hi-C and a rapidly-expanding set of other chromosome conformation protocols (3C+). Massive and heterogeneous 3C+ data require high-performance and flexible processing of sequenced reads into contact pairs. To meet these challenges, we present pairtools - a flexible suite of tools for contact extraction from sequencing data. Pairtools provides modular command-line interface (CLI) tools that can be flexibly chained into data processing pipelines. Pairtools provides both crucial core tools as well as auxiliary tools for building feature-rich 3C+ pipelines, including contact pair manipulation, filtration, and quality control. Benchmarking pairtools against popular 3C+ data pipelines shows advantages of pairtools for high-performance and flexible 3C+ analysis. Finally, pairtools provides protocol-specific tools for multi-way contacts, haplotype-resolved contacts, and single-cell Hi-C. The combination of CLI tools and tight integration with Python data analysis libraries makes pairtools a versatile foundation for a broad range of 3C+ pipelines.

show abstract

“…The ENCODE Hi-C pipeline has been developed with the Aiden lab using their Juicer suite of software tools 39 , with some updates to mapping parameters and chimeric read handling. There are essentially five steps in the pipeline (Fig 8A ); mapping (with bwa-mem) and filtering plus Pairix 40 to form a set of contacts, or pairs file. The genome is then binned into 14 resolutions (between 10bp and 2.5Mbp) by Juicer to form contact matrix (.hic) files.…”

Section: The Encode Hi-c Pipelinementioning

confidence: 99%

The ENCODE Uniform Analysis Pipelines

Hitz,

Lee,

Jolanki

et al. 2023

Preprint

7

0

View full text Add to dashboard Cite

The Encyclopedia of DNA elements (ENCODE) project is a collaborative effort to create a comprehensive catalog of functional elements in the human genome. The current database comprises more than 19000 functional genomics experiments across more than 1000 cell lines and tissues using a wide array of experimental techniques to study the chromatin structure, regulatory and transcriptional landscape of the Homo sapiens and Mus musculus genomes. All experimental data, metadata, and associated computational analyses created by the ENCODE consortium are submitted to the Data Coordination Center (DCC) for validation, tracking, storage, and distribution to community resources and the scientific community. The ENCODE project has engineered and distributed uniform processing pipelines in order to promote data provenance and reproducibility as well as allow interoperability between genomic resources and other consortia. All data files, reference genome versions, software versions, and parameters used by the pipelines are captured and available via the ENCODE Portal. The pipeline code, developed using Docker and Workflow Description Language (WDL; https://openwdl.org/) is publicly available in GitHub, with images available on Dockerhub (https://hub.docker.com), enabling access to a diverse range of biomedical researchers. ENCODE pipelines maintained and used by the DCC can be installed to run on personal computers, local HPC clusters, or in cloud computing environments via Cromwell. Access to the pipelines and data via the cloud allows small labs the ability to use the data or software without access to institutional compute clusters. Standardization of the computational methodologies for analysis and quality control leads to comparable results from different ENCODE collections - a prerequisite for successful integrative analyses.

show abstract

Pairs and Pairix: a file format and a tool for efficient storage and retrieval for Hi-C read pairs

Cited by 16 publications

References 13 publications

The ENCODE Uniform Analysis Pipelines

The ENCODE Uniform Analysis Pipelines

Pairtools: from sequencing data to chromosome contacts

The ENCODE Uniform Analysis Pipelines

Contact Info

Product

Resources

About