Discovering Statistical Models of Availability in Large Distributed Systems: An Empirical Study of SETI@home

Javadi, Bahman; Kondo, Derrick; Vincent, Jean-Marc; Anderson, David P.

doi:10.1109/tpds.2011.50

Cited by 100 publications

(52 citation statements)

References 16 publications

Supporting

Mentioning

Contrasting

Unclassified

Order By: Relevance

“…The FTA traces have been instrumental in the development of various failure models [46,55,56]. In a recent study [46], we analyzed and model the time-varying behavior of failures in large-scale distributed systems.…”

Section: Discussion: On the Current And Future Use Of The Ftamentioning

confidence: 99%

The Failure Trace Archive: Enabling the comparison of failure measurements and models of distributed systems

Javadi

Kondo

Iosup

et al. 2013

Journal of Parallel and Distributed Computing

Self Cite

View full text Add to dashboard Cite

With the increasing presence, scale, and complexity of distributed systems, resource failures are becoming an important and practical topic of computer science research. While numerous failure models and failure-aware algorithms exist, their comparison has been hampered by the lack of public failure data sets and data processing tools. To facilitate the design, validation, and comparison of fault-tolerant models and algorithms, we have created the Failure Trace Archive (FTA)-an online, public repository of failure traces collected from diverse parallel and distributed systems. In this work, we first describe the design of the archive, in particular of the standard FTA data format, and the design of a toolbox that facilitates automated analysis of trace data sets. We also discuss the use of the FTA for various current and future purposes. Second, after applying the toolbox to nine failure traces collected from distributed systems used in various application domains (e.g., HPC, Internet operation, and various online applications), we present a comparative analysis of failures in various distributed systems. underline the need for public availability of trace data from different distributed systems. Last, we show how different interpretations of the meaning of failure data can result in different conclusions for failure modeling and job scheduling in distributed systems. Our results for different interpretations show evidence that there may be a need for further revisiting existing failureaware algorithms, when applied for general rather than for domain-specific distributed systems.

show abstract

Section: Discussion: On the Current And Future Use Of The Ftamentioning

confidence: 99%

The Failure Trace Archive: Enabling the comparison of failure measurements and models of distributed systems

Javadi

Kondo

Iosup

et al. 2013

Journal of Parallel and Distributed Computing

Self Cite

View full text Add to dashboard Cite

show abstract

“…This cluster was used to process 2GB of data, 36 maps and 5 reduces. The second cluster contains 5 heterogeneous machines with 2 CPU cores each, which process capacity is drawn from a log-normal distribution according to [20] from 4.76 GFlops to 6.89 GFlops, network of 10 Mbps. This cluster is used to process 1.1 GB of data, 36 maps and 30 reduces.…”

Section: A Environment Setupmentioning

confidence: 99%

BIGhybrid -- A Toolkit for Simulating MapReduce in Hybrid Infrastructures

Anjos

Fedak

Geyer

2014

2014 International Symposium on Computer Architecture and High Performance Computing Workshop

View full text Add to dashboard Cite

Abstract-Cloud computing has increasingly been used as a platform for running large business and data processing applications. Although clouds have become highly popular, when it comes to data processing, the cost of usage is not negligible. Conversely, Desktop Grids, have been used by a plethora of projects, taking advantage of the high number of resources provided for free by volunteers. Merging cloud computing and desktop grids into hybrid infrastructure can provide a feasible low-cost solution for big data analysis. Although frameworks like MapReduce have been conceived to exploit commodity hardware, their use on hybrid infrastructure poses some challenges due to large resource heterogeneity and high churn rate. This study introduces BIGhybrid a toolkit to simulate MapReduce on hybrid environments. The main goal is to provide a framework for developers and system designers to address the issues of hybrid MapReduce. In this paper, we describe the framework which simulates the assembly of two existing middleware: BitDewMapReduce for Desktop Grids and Hadoop-BlobSeer for Cloud Computing. Experimental results included in this work demonstrate the feasibility of our approach.

show abstract

“…据此, 加州大学伯克利分校(University of California, Berkeley)空间科学实验室认为, 如果将那些被屏幕保护程序所消耗的计算机资源都用于科学计算的话, 一个原本可能需要数百年时间来完成的任务, 仅需数小时至数天就可以解决. 1999 年, 第一个分布式计算程序--外星智慧搜索程序(the Search for Extraterrestrial Intelligence program, SETI@home) [37] [36] 等都很快作出了调整以适应这种新型的异构化并行架构 [41] , 同时, 新的专门针对图形处理器而设计的分子动力学代码如 HOOMD-blue [42] 等也应运而生.…”

Section: 利用分布式闲置计算资源应对挑战unclassified

Frontiers in High-Performance, Large-Scale Molecular Dynamics. 35 Years of Molecular-Dynamics Simulations of Biological Systems

Cai¹,

Chipot²

2013

Acta Chim. Sinica

View full text Add to dashboard Cite

The main thrust of this contribution is to review applications of numerical simulations to biological systems over the past 35 years-specifically classical molecular-dynamics simulations and related preferential sampling approaches aimed at exploring selected degrees of freedom of the molecular assembly. Arguably enough, structural biology and biophysics represent one of the greatest challenges for molecular dynamics, owing to the size of the biological objects of interest and the time scales spanned by the molecular processes of the cell machinery in which these objects are prominent actors. The reader is assumed to be fully familiarized with the basic theoretical underpinnings of molecular-dynamics simulations, which will be discussed here from a biological standpoint, emphasizing how the enterprise of modeling increasingly larger molecular assemblies over physiologically relevant times has shaped the field. This review article will further show how the unbridled race to dilate both the spatial and the temporal scales, in an effort to bridge the gap between the latter, has greatly benefitted from groundbreaking advances on the hardware, computational front-notably through the development of massively parallel and dedicated architectures, as well as on the methodological, algorithmic front. The current trends in this research field, boosted by recent, cutting-edge achievements, wherein molecular dynamics has reached new frontiers, provide the basis for an introspective reflection and a prospective outlook into the future of biologically-oriented, high-performance numerical simulations. Furthermore, alternatives to brute-force molecular dynamics towards connecting time and size scales will be discussed, in particular a class of approaches relying upon the preferential sampling of judiciously chosen, important degrees of freedom of the biological object at hand. These methods, targeted primarily at providing a detailed thermodynamic picture of the molecular process at hand, can be viewed as computational tweezers designed to dissect the latter by means of a reduced set of collective variables.

show abstract

Discovering Statistical Models of Availability in Large Distributed Systems: An Empirical Study of SETI@home

Cited by 100 publications

References 16 publications

The Failure Trace Archive: Enabling the comparison of failure measurements and models of distributed systems

The Failure Trace Archive: Enabling the comparison of failure measurements and models of distributed systems

BIGhybrid -- A Toolkit for Simulating MapReduce in Hybrid Infrastructures

Frontiers in High-Performance, Large-Scale Molecular Dynamics. 35 Years of Molecular-Dynamics Simulations of Biological Systems

Contact Info

Product

Resources

About