Efficient clustered server-side data analysis workflows using SWAMP

Wang, Daniel L.; Zender, Charles S.; Jenks, Stephen

doi:10.1007/s12145-009-0021-z

Cited by 11 publications

(4 citation statements)

References 25 publications

Supporting

Mentioning

Contrasting

Unclassified

Order By: Relevance

“…An increasingly large number of workflows are available today to manage high-throughput genomics sequencing data, from basic data processing to high-quality visualization of results. Examples include shell-scripts [ 58 , 59 ], tool-specific graphical interfaces [ 60 , 61 ], and graphical workflow environments [ 62 , 63 , 64 , 65 ]. The graphical workflow environment are emerging as useful for constructing, modifying, interconnecting and executing computational genomics protocols using data processing workflows, also described as “pipelines” once the processes have been connected ( Table 2 ).…”

Section: Review Of the Current Methodologies And Tools For Ngs Dnamentioning

confidence: 99%

Next Generation Sequence Analysis and Computational Genomics Using Graphical Pipeline Workflows

Torri

Dinov

Zamanyan

et al. 2012

Genes

View full text Add to dashboard Cite

Whole-genome and exome sequencing have already proven to be essential and powerful methods to identify genes responsible for simple Mendelian inherited disorders. These methods can be applied to complex disorders as well, and have been adopted as one of the current mainstream approaches in population genetics. These achievements have been made possible by next generation sequencing (NGS) technologies, which require substantial bioinformatics resources to analyze the dense and complex sequence data. The huge analytical burden of data from genome sequencing might be seen as a bottleneck slowing the publication of NGS papers at this time, especially in psychiatric genetics. We review the existing methods for processing NGS data, to place into context the rationale for the design of a computational resource. We describe our method, the Graphical Pipeline for Computational Genomics (GPCG), to perform the computational steps required to analyze NGS data. The GPCG implements flexible workflows for basic sequence alignment, sequence data quality control, single nucleotide polymorphism analysis, copy number variant identification, annotation, and visualization of results. These workflows cover all the analytical steps required for NGS data, from processing the raw reads to variant calling and annotation. The current version of the pipeline is freely available at . These applications of NGS analysis may gain clinical utility in the near future (e.g., identifying miRNA signatures in diseases) when the bioinformatics approach is made feasible. Taken together, the annotation tools and strategies that have been developed to retrieve information and test hypotheses about the functional role of variants present in the human genome will help to pinpoint the genetic risk factors for psychiatric disorders.

show abstract

Section: Review Of the Current Methodologies And Tools For Ngs Dnamentioning

confidence: 99%

Next Generation Sequence Analysis and Computational Genomics Using Graphical Pipeline Workflows

Torri

Dinov

Zamanyan

et al. 2012

Genes

View full text Add to dashboard Cite

show abstract

“…Penelitian (Wang, Zender, & Jenks, 2009) membahas tentang teknologi yang terus memungkinkan para ilmuwan untuk mengatur arsip baru dalam pengumpulan dan produksi data, mengintensifkan kebutuhan alat skala besar untuk secara efisien memproses dan menganalisa data besar yang terus meningkat.…”

Section: A Literatur Reviewunclassified

Server-Side Processing Techniques for Optimizing the Speed of Presenting Big Data

Sulastri

Rahmatulloh

Hidayat

2019

pilar

View full text Add to dashboard Cite

Big data is the latest industry keyword to describe large volumes of structured and unstructured data that are difficult to process and analyze. Most organizations are looking for the best approach to managing and analyzing large volumes of data, especially in decision making. Large data causes the process of presenting information to be slow because all the large amounts of data must be displayed so that specific techniques are needed so that the presentation of information remains fast even though the data is already large. The website generally processes requests to the server, and then if the required data is available, the server will send all the data. This causes all processes to be based on the client-side. So that the client load becomes heavy in displaying all the data. In this study, server-side processing techniques will be applied so that all processes will be handled by the server and the data sent is not all direct but based on periodic requests from the client. The results of this study indicate the use of server-side processing techniques is more optimal. Based on the results of testing the data presentation speed comparison with server-side processing techniques 98.6% is better than client-side processing.

show abstract

“…There are several alternative approaches for high-throughput analysis of large amounts of data such as using various types of shell-scripts [11,12] and employing tool-specific graphical interfaces [13-15]. The large-scale parallelization, increased network bandwidth, need for reproducibility, and wide proliferation of efficient and robust computational and communication resources are the driving forces behind this need for automation and high-throughput analysis.…”

Section: Introductionmentioning

confidence: 99%

“…Contemporary informatics and genomic research require efficient, flexible and robust management of large heterogeneous data [ 1 , 2 ], advanced computational tools [ 3 ], powerful visualization [ 4 ], reliable hardware infrastructure [ 5 ], interoperability of computational resources [ 6 , 7 ], and provenance of data and protocols [ 8 - 10 ]. There are several alternative approaches for high-throughput analysis of large amounts of data such as using various types of shell-scripts [ 11 , 12 ] and employing tool-specific graphical interfaces [ 13 - 15 ]. The large-scale parallelization, increased network bandwidth, need for reproducibility, and wide proliferation of efficient and robust computational and communication resources are the driving forces behind this need for automation and high-throughput analysis.…”

Section: Introductionmentioning

confidence: 99%

Applications of the pipeline environment for visual informatics and genomics computations

et al. 2011

View full text Add to dashboard Cite

BackgroundContemporary informatics and genomics research require efficient, flexible and robust management of large heterogeneous data, advanced computational tools, powerful visualization, reliable hardware infrastructure, interoperability of computational resources, and detailed data and analysis-protocol provenance. The Pipeline is a client-server distributed computational environment that facilitates the visual graphical construction, execution, monitoring, validation and dissemination of advanced data analysis protocols.ResultsThis paper reports on the applications of the LONI Pipeline environment to address two informatics challenges - graphical management of diverse genomics tools, and the interoperability of informatics software. Specifically, this manuscript presents the concrete details of deploying general informatics suites and individual software tools to new hardware infrastructures, the design, validation and execution of new visual analysis protocols via the Pipeline graphical interface, and integration of diverse informatics tools via the Pipeline eXtensible Markup Language syntax. We demonstrate each of these processes using several established informatics packages (e.g., miBLAST, EMBOSS, mrFAST, GWASS, MAQ, SAMtools, Bowtie) for basic local sequence alignment and search, molecular biology data analysis, and genome-wide association studies. These examples demonstrate the power of the Pipeline graphical workflow environment to enable integration of bioinformatics resources which provide a well-defined syntax for dynamic specification of the input/output parameters and the run-time execution controls.ConclusionsThe LONI Pipeline environment http://pipeline.loni.ucla.edu provides a flexible graphical infrastructure for efficient biomedical computing and distributed informatics research. The interactive Pipeline resource manager enables the utilization and interoperability of diverse types of informatics resources. The Pipeline client-server model provides computational power to a broad spectrum of informatics investigators - experienced developers and novice users, user with or without access to advanced computational-resources (e.g., Grid, data), as well as basic and translational scientists. The open development, validation and dissemination of computational networks (pipeline workflows) facilitates the sharing of knowledge, tools, protocols and best practices, and enables the unbiased validation and replication of scientific findings by the entire community.

show abstract

Efficient clustered server-side data analysis workflows using SWAMP

Cited by 11 publications

References 25 publications

Next Generation Sequence Analysis and Computational Genomics Using Graphical Pipeline Workflows

Next Generation Sequence Analysis and Computational Genomics Using Graphical Pipeline Workflows

Server-Side Processing Techniques for Optimizing the Speed of Presenting Big Data

Applications of the pipeline environment for visual informatics and genomics computations

Contact Info

Product

Resources

About