cl-dash: rapid configuration and deployment of Hadoop clusters for bioinformatics research in the cloud

Hodor, Paul; Chawla, Amandeep; Clark, A; Neal, Lauren

doi:10.1093/bioinformatics/btv553

Cited by 11 publications

(3 citation statements)

References 6 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The new spectra-cluster algorithm was specifically developed using the Apache Hadoop framework13, 14 to reach two main goals: 1) to increase spectrum clustering accuracy and 2) to be scalable to handle the exponential data increase in PRIDE Archive. To increase spectrum clustering accuracy, based on the proportion of incorrectly clustered spectra, we developed a novel method to assess the similarity between two spectra: instead of the commonly used normalized dot product we employed a probabilistic scoring approach similar to that of the spectrum library search engine Pepitome15 (Online Methods).…”

Section: Resultsmentioning

confidence: 99%

Recognizing millions of consistently unidentified spectra across hundreds of shotgun proteomics datasets

et al. 2016

View full text Add to dashboard Cite

Mass spectrometry (MS) is the main technology used in proteomics approaches. However, on average 75% of spectra analysed in an MS experiment remain unidentified. We propose to use spectrum clustering at a large-scale to shed a light on these unidentified spectra. PRoteomics IDEntifications database (PRIDE) Archive is one of the largest MS proteomics public data repositories worldwide. By clustering all tandem MS spectra publicly available in PRIDE Archive, coming from hundreds of datasets, we were able to consistently characterize three distinct groups of spectra: 1) incorrectly identified spectra, 2) spectra correctly identified but below the set scoring threshold, and 3) truly unidentified spectra. Using a multitude of complementary analysis approaches, we were able to identify less than 20% of the consistently unidentified spectra. The complete spectrum clustering results are available through the new version of the PRIDE Cluster resource (http://www.ebi.ac.uk/pride/cluster). This resource is intended, among other aims, to encourage and simplify further investigation into these unidentified spectra.

show abstract

Section: Resultsmentioning

confidence: 99%

Recognizing millions of consistently unidentified spectra across hundreds of shotgun proteomics datasets

et al. 2016

View full text Add to dashboard Cite

show abstract

“…erefore, we have important significance for the study of job scheduling algorithms. e task scheduling algorithm is considered a complex process because it must make full use of the available resources to perform a large number of tasks [11]. It simplifies the file consistency model when it is stored.…”

Section: Introductionmentioning

confidence: 99%

Load Balancing Algorithms for Hadoop Cluster in Unbalanced Environment

Wang

2022

Computational Intelligence and Neuroscience

View full text Add to dashboard Cite

Considering that in the process of job scheduling, the cluster load should be prebalanced rather than remedied when the load is seriously unbalanced; therefore, in this paper, the task scheduling flow of the Hadoop cluster is analyzed deeply. On the Hadoop platform, a self-dividing algorithm is proposed for load balancing. An intelligent optimization algorithm is used to solve load balance. A dynamic feedback load balancing scheduling method is proposed from the point of view of task scheduling. In order to solve the shortcoming of the fair scheduling algorithm, this paper proposes two ways to improve the resource utilization and overall performance of Hadoop. When the mapping task is completed and the tasks to be reduced are assigned, the task assignment is based on the performance of the nodes to be reduced. It gives full play to the advantages of the ant colony algorithm and the hive colony algorithm so that the fusion algorithm can better deal with load balance. Then, three existing scheduling algorithms are introduced in detail: single queue scheduling, capacity scheduling, and fair scheduling. On this basis, an improved task scheduling strategy based on genetic algorithm is proposed to allocate and execute application tasks to reduce task completion time. The experiment verifies the effectiveness of the algorithm. The LBNP algorithm greatly improves the efficiency of reducing task execution and job execution. The delay capacity scheduling algorithm can ensure that most tasks can achieve localization scheduling, improve resource utilization, improve load balance, and speed up job completion time.

show abstract

“…Distributed computing has mainly been selected as the method for cloud computing. With the development of grid computing, computation on the cloud by Apache Hadoop has been conducted extensively [3,5,7,8] and support tools for constructing Hadoop clusters on the cloud have been established [17]. However, while Hadoop/MapReduce can easily construct a distributed task calculation environment, it is versatile and therefore contains an excessive amount of functions.…”

Section: Introductionmentioning

confidence: 99%

High-performance cloud computing for exhaustive protein-protein docking

Ohue¹,

Aoyama²,

Akiyama³

2020

Preprint

View full text Add to dashboard Cite

Public cloud computing environments, such as Amazon AWS, Microsoft Azure, and the Google Cloud Platform, have achieved remarkable improvements in computational performance in recent years, and are also expected to be able to perform massively parallel computing. As the cloud enables users to use thousands of CPU cores and GPU accelerators casually, and various software types can be used very easily by cloud images, the cloud is beginning to be used in the field of bioinformatics. In this study, we ported the original protein-protein interaction prediction (protein-protein docking) software, MEGADOCK, into Microsoft Azure as an example of an HPC cloud environment. A cloud parallel computing environment with up to 1,600 CPU cores and 960 GPUs was constructed using four CPU instance types and two GPU instance types, and the parallel computing performance was evaluated. Our MEGADOCK on Azure system showed a strong scaling value of 0.93 for the CPU instance when H16 instance with 100 instances were used compared to 50, and a strong scaling value of 0.89 for the GPU instance when NC24 instance with 20 were used compared to 5. Moreover, the results of the usage fee and total computation time supported that using a GPU instance reduced the computation time of MEGADOCK and the cloud usage fee required for the computation. The developed environment deployed on the cloud is highly portable, making it suitable for applications in which an on-demand and large-scale HPC environment is desirable.

show abstract

cl-dash: rapid configuration and deployment of Hadoop clusters for bioinformatics research in the cloud

Cited by 11 publications

References 6 publications

Recognizing millions of consistently unidentified spectra across hundreds of shotgun proteomics datasets

Recognizing millions of consistently unidentified spectra across hundreds of shotgun proteomics datasets

Load Balancing Algorithms for Hadoop Cluster in Unbalanced Environment

High-performance cloud computing for exhaustive protein-protein docking

Contact Info

Product

Resources

About