Computing low latency batches with unreliable workers in volunteer computing environments

Large scale distributed systems are composed of many thousands of computing units. Today's examples of such systems are grid, volunteer and cloud computing platforms. Generally, their analyses are done through monitoring tools that gather resource information like processor or network utilization, providing high-level statistics and basic resource usage traces. Such approaches are recognized as rather scalable but are unfortunately often insufficient to detect or fully understand unexpected behavior. In this paper, we investigate the use of more detailed tracing techniques -commonly used in parallel computing-in distributed systems. Finely analyzing the behavior of such systems comprising thousands of resources over several months may seem infeasible. Yet, we show that the resulting trace can be analyzed using tools that enable to easily zoom in and out on selected area of space and time. We use the BOINC volunteer computing system as a basis of this study. Since detailed activity traces of the BOINC clients are not available yet, we rely instead on traces obtained through a BOINC simulator developed with the SimGrid toolkit and which uses as input real availability trace files from the Seti@Home BOINC project. We show that the analysis of such detailed resource utilization traces provides several non-trivial insights about the whole system and enables the discovery of unexpected behavior.

show abstract

“…Such behavior was already anticipated and optimized in other works [9]. What we would like to observe here is the shape of this slow-start.…”

Section: Response Timesupporting

confidence: 70%

Multi-scale analysis of large distributed computing systems

Schnorr

Legrand

Vincent

2011

Proceedings of the Third International Workshop on Large-Scale System and Application Performance

View full text Add to dashboard Cite

show abstract

“…Модель PULL назначения заданий описана в [55] и нацелена на минимизацию числа нарушений сроков возврата результатов. Для предложенного алгоритма доказан нулевой уровень числа таких на-рушений при полной надежности вычислительных узлов.…”

Section: исследовательские статьиunclassified

A survey of task scheduling in Desktop Grid

Chernov¹,

Ivashko²,

Nikitina³

2017

PSTA

View full text Add to dashboard Cite

“…Resource availability is modeled with statistical distributions in [12,17,25,26]. In [12], Hein et al model unreliable hosts with a Weibull distribution.…”

Section: Modeling Grid and Vc Projectsmentioning

confidence: 99%

“…In [12], Hein et al model unreliable hosts with a Weibull distribution. In [25], Wolski et al model resource capabilities, dynamic behavior, availability, failure, and connectivity for grid and global computing systems with Weibull and hyper exponential distributions.…”

Section: Modeling Grid and Vc Projectsmentioning

confidence: 99%

Performance Prediction and Analysis of BOINC Projects: An Empirical Study with EmBOINC

2009

View full text Add to dashboard Cite

Middleware systems for volunteer computing convert a set of computers that is large and diverse (in terms of hardware, software, availability, reliability, and trustworthiness) into a unified computing resource. This involves a number of scheduling policies and parameters, which have a large impact on the throughput and other performance metrics. How can we study and refine these policies? Experimentation in the context of a working project is problematic, and it is difficult to accurately model complex middleware in a conventional simulator. Instead, we use an approach in which the policies being studied are "emulated", using parts of the actual middleware. In this paper we describe EmBOINC, an emulator based on the BOINC middleware system. EmBOINC simulates a population of volunteered clients (including heterogeneity, churn, availability, and reliability) and emulates the BOINC server components. After describing the design of EmBOINC and its validation, we present three case studies in which the impact of different scheduling policies are quantified in terms of throughput, latency, and starvation metrics.

show abstract

Computing low latency batches with unreliable workers in volunteer computing environments

Cited by 29 publications

References 14 publications

Multi-scale analysis of large distributed computing systems

Multi-scale analysis of large distributed computing systems

A survey of task scheduling in Desktop Grid

Performance Prediction and Analysis of BOINC Projects: An Empirical Study with EmBOINC

Contact Info

Product

Resources

About