Best practices for the deployment and management of production HPC clusters

McLay, Robert; Schulz, Karl; Barth, William L.; Minyard, Tommy

doi:10.1145/2063348.2063360

Cited by 39 publications

(28 citation statements)

References 3 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…To facilitate using SPEAQeasy we provide Docker containers for both the software and annotation files and a SPEAQeasy configuration file for such environments. For SGE or other clusters, SPEAQeasy can also use lmod [ 50 ] software modules such as the one we provide for the JHPCE SGE cluster ( https://jhpce.jhu.edu/) . In order to use SPEAQeasy in a particular computing environment, identify the example configuration file (Implementation: configuration; Additional file 3 :Table S1) that most resembles the setup, make a copy and edit accordingly.…”

Section: Resultsmentioning

confidence: 99%

SPEAQeasy: a scalable pipeline for expression analysis and quantification for R/bioconductor-powered RNA-seq analyses

Eagles

Burke

Leonard³

et al. 2021

BMC Bioinformatics

View full text Add to dashboard Cite

Background RNA sequencing (RNA-seq) is a common and widespread biological assay, and an increasing amount of data is generated with it. In practice, there are a large number of individual steps a researcher must perform before raw RNA-seq reads yield directly valuable information, such as differential gene expression data. Existing software tools are typically specialized, only performing one step–such as alignment of reads to a reference genome–of a larger workflow. The demand for a more comprehensive and reproducible workflow has led to the production of a number of publicly available RNA-seq pipelines. However, we have found that most require computational expertise to set up or share among several users, are not actively maintained, or lack features we have found to be important in our own analyses. Results In response to these concerns, we have developed a Scalable Pipeline for Expression Analysis and Quantification (SPEAQeasy), which is easy to install and share, and provides a bridge towards R/Bioconductor downstream analysis solutions. SPEAQeasy is portable across computational frameworks (SGE, SLURM, local, docker integration) and different configuration files are provided (http://research.libd.org/SPEAQeasy/). Conclusions SPEAQeasy is user-friendly and lowers the computational-domain entry barrier for biologists and clinicians to RNA-seq data processing as the main input file is a table with sample names and their corresponding FASTQ files. The goal is to provide a flexible pipeline that is immediately usable by researchers, regardless of their technical background or computing environment.

show abstract

Section: Resultsmentioning

confidence: 99%

SPEAQeasy: a scalable pipeline for expression analysis and quantification for R/bioconductor-powered RNA-seq analyses

Eagles

Burke

Leonard³

et al. 2021

BMC Bioinformatics

View full text Add to dashboard Cite

show abstract

“…Spack provides advanced software management utilities (McLay et al, 2011) beyond library installation. It allows multiple environments to coexist and provides an easy way to switch between them (via the "spack env" command).…”

Section: Appendix B Approaches To Install Hpc Software Libraries On mentioning

confidence: 99%

Enabling high-performance cloud computing for Earth science modeling on over a thousand cores: application to the GEOS-Chem atmospheric chemistry model

Zhuang¹,

Jacob²,

Lin³

et al. 2020

Preprint

View full text Add to dashboard Cite

Cloud computing platforms can facilitate the use of Earth science models by providing immediate access to fully configured software, massive computing power, and large input datasets. However, slow inter-node communication performance has previously discouraged the use of cloud platforms for massively parallel simulations. Here we show that recent advances in the network performance on the Amazon Web Services (AWS) cloud enable efficient model simulations with over a thousand cores. The choices of Message Passing Interface (MPI) library configuration and inter-node communication protocol are critical to this success. Application to the GEOS-Chem model of atmospheric chemistry at global 50 km horizontal resolution shows efficient scaling up to at least 1152 cores, with performance and cost comparable to the NASA Pleiades supercomputing cluster.

show abstract

“…The comparison and auxiliary tools are placed in a “tools” directory. We have also included examples of submission scripts in the “hpc” directory, with PBS [18, 19] and SLURM [20, 21] examples of submission scripts, so that users with access to HPC resources can take advantage of them. Additionally, we have included a “datasets” directory containing scripts that allow the user to download the data used in these analyses.…”

Section: Racs Pipeline Implementationmentioning

confidence: 99%

RACS: rapid analysis of ChIP-Seq data for contig based genomes

et al. 2019

View full text Add to dashboard Cite

BackgroundChromatin immunoprecipitation coupled to next generation sequencing (ChIP-Seq) is a widely-used molecular method to investigate the function of chromatin-related proteins by identifying their associated DNA sequences on a genomic scale. ChIP-Seq generates large quantities of data that is difficult to process and analyze, particularly for organisms with a contig-based sequenced genomes that typically have minimal annotation on their associated set of genes other than their associated coordinates primarily predicted by gene finding programs. Poorly annotated genome sequence makes comprehensive analysis of ChIP-Seq data difficult and as such standardized analysis pipelines are lacking.ResultsWe present a one-stop computational pipeline, “Rapid Analysis of ChIP-Seq data” (RACS), that utilizes traditional High-Performance Computing (HPC) techniques in association with open source tools for processing and analyzing raw ChIP-Seq data. RACS is an open source computational pipeline available from any of the following repositories https://bitbucket.org/mjponce/RACS or https://gitrepos.scinet.utoronto.ca/public/?a=summary&p=RACS. RACS is particularly useful for ChIP-Seq in organisms with contig-based genomes that have poor gene annotation to aid protein function discovery.To test the performance and efficiency of RACS, we analyzed ChIP-Seq data previously published in a model organism Tetrahymena thermophila which has a contig-based genome. We assessed the generality of RACS by analyzing a previously published data set generated using the model organism Oxytricha trifallax, whose genome sequence is also contig-based with poor annotation.ConclusionsThe RACS computational pipeline presented in this report is an efficient and reliable tool to analyze genome-wide raw ChIP-Seq data generated in model organisms with poorly annotated contig-based genome sequence. Because RACS segregates the found read accumulations between genic and intergenic regions, it is particularly efficient for rapid downstream analyses of proteins involved in gene expression.

show abstract

Best practices for the deployment and management of production HPC clusters

Cited by 39 publications

References 3 publications

SPEAQeasy: a scalable pipeline for expression analysis and quantification for R/bioconductor-powered RNA-seq analyses

SPEAQeasy: a scalable pipeline for expression analysis and quantification for R/bioconductor-powered RNA-seq analyses

Enabling high-performance cloud computing for Earth science modeling on over a thousand cores: application to the GEOS-Chem atmospheric chemistry model

RACS: rapid analysis of ChIP-Seq data for contig based genomes

Contact Info

Product

Resources

About