Gumshoe: Diagnosing Performance Problems in Replicated File-Systems

Kavulya, Soila; Gandhi, Rajeev; Narasimhan, Priya

doi:10.1109/srds.2008.35

Cited by 4 publications

(4 citation statements)

References 19 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The rationale of their approach is based on the observation that there is an obvious difference between the behaviors of fault-free and faulty nodes. Kavulya et al [9] and Lan et al [8] proposed the similar approaches to detect performance problems in replicated file systems and a cluster system, respectively. However, the validation of these approaches is based on a strong assumption of homogeneous hardware and workloads, which may only hold in a few cases.…”

Section: Related Workmentioning

confidence: 99%

Design and Evaluation of an Online Anomaly Detector for Distributed Storage Systems

Chen¹,

He²,

He³

et al. 2011

JSW

View full text Add to dashboard Cite

Performance problems, which may stem from different system components, such as network, memory, and storage devices, are difﬁcult to diagnose and isolate in distributed storage systems. In this paper, we present a performance anomaly detector which is able to efﬁciently detect performance anomaly and accurately identify the faulty sources in a system node of a distributed storage system. Our method exploits the stable relationship between workloads and system resource statistics to detect the performance anomaly and identify faulty sources which cause the performance anomaly in the system. Our experimental results demonstrate the efﬁciency and accuracy of the proposed performance anomaly detector

show abstract

Section: Related Workmentioning

confidence: 99%

Design and Evaluation of an Online Anomaly Detector for Distributed Storage Systems

Chen¹,

He²,

He³

et al. 2011

JSW

View full text Add to dashboard Cite

show abstract

“…The rationale of their approach is based on the observation that there is an obvious difference between the behaviors of fault-free and faulty nodes. Kavulya et al [10] and Lan et al [9] proposed the similar approaches to detect performance problems in replicated file systems and a cluster system, respectively. However, the validation of these approaches is based on a strong assumption of homogeneous hardware and workloads, which may only hold in a few cases.…”

Section: Related Workmentioning

confidence: 99%

An Online Performance Anomaly Detector in Cluster File Systems

Chen

et al. 2010

2010 3rd International Symposium on Parallel Architectures, Algorithms and Programming

View full text Add to dashboard Cite

Abstract-Performance problems, which can stem from different system components, such as network, memory, and storage devices, are difficult to diagnose and isolate in a cluster file system. In this paper, we present an online performance anomaly detector which is able to efficiently detect performance anomaly and accurately identify the faulty sources in a system node of a cluster file system. Our method exploits the stable relationship between workloads and system resource statistics to detect the performance anomaly and identify faulty sources which cause the performance anomaly in the system. Our preliminary experimental results demonstrate the efficiency and accuracy of the proposed performance anomaly detector.

show abstract

“…Textual console log analysis in high performance computing [6], [7] is more flexible, but maintaining such logs may be impractical in high-volume systems where transactions are very short, time-sensitive, and rapid; textual logs would be immense -difficult to output, store and retrieve. Finally, some unsupervised approaches [8], [9] rely on domain insights and system knowledge, and therefore have limited applicability.…”

Section: Introductionmentioning

confidence: 99%

“…Recent approaches to the monitoring problem [3], [9], [10] focus on early detection and handling of performance problems, or latent faults. These are outliers -machine behaviors that could indicate a fault yet fly under the radar of monitoring systems because they are not acute enough, or were not anticipated by maintenance engineers.…”

Section: Introductionmentioning

confidence: 99%

Communication-Efficient Distributed Variance Monitoring and Outlier Detection for Multivariate Time Series

Gabel

Schuster

Keren

2014

2014 IEEE 28th International Parallel and Distributed Processing Symposium

View full text Add to dashboard Cite

Modern scale-out services are comprised of thousands of individual machines, which must be continuously monitored for unexpected failures. One recent approach to monitoring is latent fault detection, an adaptive statistical framework for scale-out, load-balanced systems. By periodically measuring hundreds of performance metrics and looking for outlier machines, it attempts to detect subtle problems such as misconfigurations, bugs, and malfunctioning hardware, before they manifest as machine failures. Previous work on a large, real-world Web service has shown that many failures are indeed preceded by such latent faults. Latent fault detection is an offline framework with large bandwidth and processing requirements. Each machine must send all its measurements to a centralized location, which is prohibitive in some settings and requires data-parallel processing infrastructure. In this work we adapt the latent fault detector to provide an online, communication-and computationreduced version. We utilize stream processing techniques to trade accuracy for communication and computation. We first describe a novel communication-efficient online distributed variance monitoring algorithm that provides a continuous estimate of the global variance within guaranteed approximation bounds. Using the variance monitor, we provide an online distributed outlier detection framework for nonstationary multivariate time series common in scale-out systems. The adapted framework reduces data size and central processing cost by processing the data in situ, making it usable in wider settings. Like the original framework, our adaptation admits different comparison functions, supports non-stationary data, and provides statistical guarantees on the rate of false positives. Simulations on logs from a production system show that we are able to reduce bandwidth by an order of magnitude, with below 1% error compared to the original algorithm.

show abstract

Gumshoe: Diagnosing Performance Problems in Replicated File-Systems

Cited by 4 publications

References 19 publications

Design and Evaluation of an Online Anomaly Detector for Distributed Storage Systems

Design and Evaluation of an Online Anomaly Detector for Distributed Storage Systems

An Online Performance Anomaly Detector in Cluster File Systems

Communication-Efficient Distributed Variance Monitoring and Outlier Detection for Multivariate Time Series

Contact Info

Product

Resources

About