Dryad

Isard, Michael; Budiu, Mihai; Yu, Yuan; Birrell, Andrew; Fetterly, Dennis

doi:10.1145/1272998.1273005

Cited by 525 publications

(21 citation statements)

References 35 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…MapReduce and provides developers with a convenient paradigm for modeling complex applications that require multiple execution steps. Dryad (Isard et al 2007) is based on dataflow graph processing-based programming model that supports scalable and distributed programming applications. Another application framework that allows execution of a complex DAG created from tasks is Apache Tez 4 .…”

Section: Dag Processingmentioning

confidence: 99%

Big Data Computing Using Cloud-Based Technologies

Khan¹,

Shakil²,

Alam³

2017

Networks of the Future

View full text Add to dashboard Cite

The excessive amounts of data generated by devices and Internet-based sources at a regular basis constitute, big data. This data can be processed and analyzed to develop useful applications for specific domains. Several mathematical and data analytics techniques have found use in this sphere. This has given rise to the development of computing models and tools for big data computing. However, the storage and processing requirements are overwhelming for traditional systems and technologies. Therefore, there is a need for infrastructures that can adjust the storage and processing capability in accordance with the changing data dimensions.Cloud Computing serves as a potential solution to this problem. However, big data computing in the cloud has its own set of challenges and research issues. This chapter surveys the big data concept, discusses the mathematical and data analytics techniques that can be used for big data and gives taxonomy of the existing tools, frameworks and platforms available for different big data computing models. Besides this, it also evaluates the viability of cloud-based big data computing, examines existing challenges and opportunities, and provides future research directions in this field.

show abstract

Section: Dag Processingmentioning

confidence: 99%

Big Data Computing Using Cloud-Based Technologies

Khan¹,

Shakil²,

Alam³

2017

Networks of the Future

View full text Add to dashboard Cite

show abstract

“…The software is known as Cloudburst and it is implemented on top of the Hadoop platform for the analysis of next generation sequencing data. Performance comparison studies have been conducted between a message passing interface (MPI) [51], Dryad [52], and a Hadoop MapReduce programming framework for measuring relative performance using three bioinformatics applications [53]. BLAST and gene set enrichment analysis (GSEA) algorithms have been implemented in Hadoop [54] for streaming computation on large data sets and a multi-pass computation on relatively small datasets.…”

Section: Reviewmentioning

confidence: 99%

Applications of the MapReduce programming framework to clinical big data analysis: current landscape and future trends

2014

View full text Add to dashboard Cite

The emergence of massive datasets in a clinical setting presents both challenges and opportunities in data storage and analysis. This so called “big data” challenges traditional analytic tools and will increasingly require novel solutions adapted from other fields. Advances in information and communication technology present the most viable solutions to big data analysis in terms of efficiency and scalability. It is vital those big data solutions are multithreaded and that data access approaches be precisely tailored to large volumes of semi-structured/unstructured data.The MapReduce programming framework uses two tasks common in functional programming: Map and Reduce. MapReduce is a new parallel processing framework and Hadoop is its open-source implementation on a single computing node or on clusters. Compared with existing parallel processing paradigms (e.g. grid computing and graphical processing unit (GPU)), MapReduce and Hadoop have two advantages: 1) fault-tolerant storage resulting in reliable data processing by replicating the computing tasks, and cloning the data chunks on different computing nodes across the computing cluster; 2) high-throughput data processing via a batch processing framework and the Hadoop distributed file system (HDFS). Data are stored in the HDFS and made available to the slave nodes for computation.In this paper, we review the existing applications of the MapReduce programming framework and its implementation platform Hadoop in clinical big data and related medical health informatics fields. The usage of MapReduce and Hadoop on a distributed system represents a significant advance in clinical big data processing and utilization, and opens up new opportunities in the emerging era of big data analytics. The objective of this paper is to summarize the state-of-the-art efforts in clinical big data analytics and highlight what might be needed to enhance the outcomes of clinical big data analytics tools. This paper is concluded by summarizing the potential usage of the MapReduce programming framework and Hadoop platform to process huge volumes of clinical data in medical health informatics related fields.

show abstract

“…In Stratosphere, DDP patterns are expressed using the Parallelization Contract (PACTs) programming model [7], and the PACT compiler converts PACT programs into jobs for the Nephele execution engine [19]. DryadLINQ [20] is a high-level programming language consisting of .NET constructs for operating on datasets and runs on Dryad [21], a distributed data flow execution engine. Other available DDP execution engines include Cloud MapReduce, Phoenix and MapReduce-MPI.…”

Section: Related Workmentioning

confidence: 99%

A Framework for Distributed Data-Parallel Execution in the Kepler Scientific Workflow System

Wang

Crawl

Altıntaş

2012

Procedia Computer Science

View full text Add to dashboard Cite

Distributed Data-Parallel (DDP) patterns such as MapReduce have become increasingly popular as solutions to facilitate data-intensive applications, resulting in a number of systems supporting DDP workflows. Yet, applications or workflows built using these patterns are usually tightly-coupled with the underlying DDP execution engine they select. We present a framework for distributed data-parallel execution in the Kepler scientific workflow system that enables users to easily switch between different DDP execution engines. We describe a set of DDP actors based on DDP patterns and directors for DDP workflow executions within the presented framework. We demonstrate how DDP workflows can be easily composed in the Kepler graphic user interface through the reuse of these DDP actors and directors and how the generated DDP workflows can be executed in different distributed environments. Via a bioinformatics usecase, we discuss the usability of the proposed framework and validate its execution scalability.

show abstract

Dryad

Cited by 525 publications

References 35 publications

Big Data Computing Using Cloud-Based Technologies

Big Data Computing Using Cloud-Based Technologies

Applications of the MapReduce programming framework to clinical big data analysis: current landscape and future trends

A Framework for Distributed Data-Parallel Execution in the Kepler Scientific Workflow System

Contact Info

Product

Resources

About