Mining for statistical models of availability in large-scale distributed systems: An empirical study of SETI@home

Javadi, Bahman; Kondo, Derrick; Vincent, Jean-Marc; Anderson, David P.

doi:10.1109/mascot.2009.5367061

Cited by 60 publications

(46 citation statements)

References 16 publications

Supporting

Mentioning

Contrasting

Unclassified

Order By: Relevance

“…The scheduling discipline of each queue is rst-come-rst-served (FCFS). This model is common in grid and volunteer computing [31,1,22], and supercomputing [30,19]. To analyze the addedvalue of letting the router exploit its bu er, we introduce the Price of Forgetting (PoF), which we de ne as the ratio between the minimum response time achievable by a memoryless broker with respect to its counterpart with memory.…”

Section: Our Contributionmentioning

confidence: 99%

“…Our routing policy requires the solution of a convex optimization problem, and numerical results show that systems with thousands of queues can be solved in a few seconds. Our scheme has been successfully implemented in the context of a real-world volunteer-computing system [22].…”

Section: Our Contributionmentioning

confidence: 99%

“…The expression (22) and (23) follows by substitution of (57) in g(x) and in U i = 1 − 0.5σ i , respectively.…”

Section: Proof Of Theoremmentioning

confidence: 99%

See 2 more Smart Citations

The price of forgetting in parallel and non-observable queues

Anselmi¹,

Gaujal²

2011

Performance Evaluation

View full text Add to dashboard Cite

We consider a broker-based network of non-observable parallel queues and analyze the minimum expected response time and the optimal routing policy when the broker has the memory of its previous routing decisions. We provide lower bounds on the minimum response time by means of convex programming that are tight, as follows by a numerical comparison with a proposed routing scheme. The Price of Forgetting (PoF), the ratio between the minimum response times achieved by a probabilistic broker and a broker with memory, is shown to be unbounded or arbitrarily close to one depending on the coe cient of variation of the service time distributions. In the case of exponential service times, the PoF is bounded from above by two, which is tight in heavy-tra c, and independent of the network size and heterogeneity. These properties yield a simple engineering product-form approximating tightly the minimum response time. Finally, we put our results in the context of game theory revisiting the Price of Anarchy (PoA) of parallel queues: It can be decomposed into the product of the PoA achieved by a probabilistic broker (already well understood) and the PoF. IntroductionIn the context of *-computing and manufacturing systems, users submit jobs without knowing which machine will handle their execution. A central broker is usually in charge of distributing incoming jobs to a set of resources to optimize some utility or cost function. The mean response time (simply response time in the following), the expected time it takes for a job to join the system and return to its issuer, is an important metric that is usually taken into account to achieve a better exploitation of resources and improve the average quality of service.The problem of nding the routing strategy (or policy) that minimizes response time and the resulting response time itself are well-known problem in queueing theory and the literature is overwhelming. There are two main classes of routing strategies: Closed-loop policies, where the broker knows the state of each resource (number of jobs), and open-loop policies, where the broker does not know their state. Computing the best closed-loop policy is known to be a hard problem and a lot of work has been devoted to the computation of good approximations; see, e.g., [9,27, 33] and the references therein.The focus of this paper is on the open-loop case, and more precisely on o -line policies. Indeed, the state of the queues may not to be known on-line to the broker because of a number of reasons whose e ects become worse and worse as the network size increases: i) Communicating the system state to the broker increases the network load, ii) the information received by the broker can be out of date, and iii) the synchronization that results from knowing the system state can degrade the performance [26]. A celebrated case is when the broker only knows the service time distributions of the network resources (or queues in the following) and dispatches jobs to queues according to an i.i.d. probabilistic law. The probabili...

show abstract

Section: Our Contributionmentioning

confidence: 99%

Section: Our Contributionmentioning

confidence: 99%

See 1 more Smart Citation

The price of forgetting in parallel and non-observable queues

Anselmi¹,

Gaujal²

2011

Performance Evaluation

View full text Add to dashboard Cite

show abstract

“…In [10], the authors studied classification of the availability of CPUs in a massive distributed system through the use of clustering techniques. In their analysis, they excluded all nodes with nonrandom behavior to focus on emergent patterns across nodes whose behaviors are independent and identically distributed.…”

Section: Related Workmentioning

confidence: 99%

A Characterization of Node Uptime Distributions in the PlanetLab Test Bed

Verespej

Pasquale

2011

2011 IEEE 30th International Symposium on Reliable Distributed Systems

View full text Add to dashboard Cite

Abstract-In this paper, we study nodes from the PlanetLab test bed to form a model of their uptime behavior. By applying clustering techniques to over a year's worth of availability data for the nodes, we identify six uptime distributions, each exhibiting unique characteristics shared by the nodes within it. The behavioral patterns exhibited by these groups, combined with the behaviors exhibited by the aggregate across the system, provide useful information for researchers designing applications that are run or tested on PlanetLab.

show abstract

“…Pinheiro et al conducted a comprehensive statistical study on failure trends in a large disk drive population [43]. Javadi et al discovered statistical models of host availability in a largescale distributed system SETI@home [44]. Different from the above work, this work focuses on quantifying the predictability of real-world system anomalies.…”

Section: Related Workmentioning

confidence: 99%

On Predictability of System Anomalies in Real World

Tan

2010

2010 IEEE International Symposium on Modeling, Analysis and Simulation of Computer and Telecommunication Systems

View full text Add to dashboard Cite

Abstract-As computer systems become increasingly complex, system anomalies have become major concerns in system management. In this paper, we present a comprehensive measurement study to quantify the predictability of different system anomalies. Online anomaly prediction allows the system to foresee impending anomalies so as to take proper actions to mitigate anomaly impact. Our anomaly prediction approach combines feature value prediction with statistical classification methods. We conduct extensive measurement study to investigate anomalous behavior of three systems in the real world: PlanetLab, SMART hard drive data, and IBM System S. We observe that real world system anomalies do exhibit predictability, which can be predicted with high accuracy and significant lead time.

show abstract

Mining for statistical models of availability in large-scale distributed systems: An empirical study of SETI@home

Cited by 60 publications

References 16 publications

The price of forgetting in parallel and non-observable queues

The price of forgetting in parallel and non-observable queues

A Characterization of Node Uptime Distributions in the PlanetLab Test Bed

On Predictability of System Anomalies in Real World

Contact Info

Product

Resources

About