VScope: Middleware for Troubleshooting Time-Sensitive Data Center Applications

Wang, Chengwei; Rayan, Infantdani Abel; Eisenhauer, Greg; Schwan, Karsten; Talwar, Vanish; Wolf, Matthew; Huneycutt, Chad Marcus

doi:10.1007/978-3-642-35170-9_7

Cited by 24 publications

(7 citation statements)

References 21 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…It reflects the actual CPU time allocated to the virtual node and can effectively calibrate the configuration of virtual hardware according to the experienced interferences. Ant uses the k-means [26] clustering algorithm to classify virtual nodes into configuration groups. In this work, we only classify virtual nodes at the beginning of self-tuning process.…”

Section: Forming Self-tuning Groupsmentioning

confidence: 99%

Improving MapReduce performance in heterogeneous environments with adaptive task tuning

Cheng

Rao

Guo

et al. 2014

Proceedings of the 15th International Middleware Conference on - Middleware '14

View full text Add to dashboard Cite

The deployment of MapReduce in datacenters and clouds present several challenges in achieving good job performance. Compared to in-house dedicated clusters, datacenters and clouds often exhibit significant hardware and performance heterogeneity due to continuous server replacement and multitenant interferences. As most Mapreduce implementations assume homogeneous clusters, heterogeneity can cause significant load imbalance in task execution, leading to poor performance and low cluster utilizations. Despite existing optimizations on task scheduling and load balancing, MapReduce still performs poorly on heterogeneous clusters.In this paper, we find that the homogeneous configuration of tasks on heterogeneous nodes can be an important source of load imbalance and thus cause poor performance. Tasks should be customized with different settings to match the capabilities of heterogeneous nodes. To this end, we propose an adaptive task tuning approach, Ant, that automatically finds the optimal settings for individual tasks running on different nodes. Ant works best for large jobs with multiple rounds of map task execution. It first configures tasks with randomly selected configurations and gradually improves tasks settings by reproducing the settings from best performing tasks and discarding poor performing configurations. To accelerate task tuning and avoid trapping in local optimum, Ant uses genetic functions during task configuration. Experimental results on a heterogeneous cluster and a virtual cluster with varying hardware capabilities show that Ant improves the average job completion time by 23%, 11%, and 16% compared to stock Hadoop, customized Hadoop with industry recommendations, and a profiling-based configuration approach, respectively.

show abstract

Section: Forming Self-tuning Groupsmentioning

confidence: 99%

Improving MapReduce performance in heterogeneous environments with adaptive task tuning

Cheng

Rao

Guo

et al. 2014

Proceedings of the 15th International Middleware Conference on - Middleware '14

View full text Add to dashboard Cite

show abstract

“…Another popular scenario for interactive cube exploration is the management of cloud infrastructure. Tools such as VScope [23] have been developed for datacenter monitoring. For each setup, a handful of operations personnel manage tens of thousands of nodes, each with multiple virtual machines.…”

Section: Scenariomentioning

confidence: 99%

Combining user interaction, speculative query execution and sampling in the DICE system

et al. 2014

View full text Add to dashboard Cite

The interactive exploration of data cubes has become a popular application, especially over large datasets. In this paper, we present DICE, a combination of a novel frontend query interface and distributed aggregation backend that enables interactive cube exploration. DICE provides a convenient, practical alternative to the typical offline cube materialization strategy by allowing the user to explore facets of the data cube, trading off accuracy for interactive response-times, by sampling the data. We consider the time spent by the user perusing the results of their current query as an opportunity to execute and cache the most likely followup queries. The frontend presents a novel intuitive interface that allows for sampling-aware aggregations, and encourages interaction via our proposed faceted model. The design of our backend is tailored towards the low-latency user interaction at the frontend, and viceversa. We discuss the synergistic design behind both the frontend user experience and the backend architecture of DICE; and, present a demonstration that allows the user to fluidly interact with billiontuple datasets within sub-second interactive response times.

show abstract

“…In particular, VScope [32] and Monalytics [31] allow monitoring teams to consider the relationships between sensors for root cause finding. However, such middleware rely on manual rule-based configuration thereby requiring deep domain knowledge.…”

Section: Related Workmentioning

confidence: 99%

Root cause detection in a service-oriented architecture

Kim

Sumbaly

Shah

2013

Proceedings of the ACM SIGMETRICS/international Conference on Measurement and Modeling of Computer Systems

View full text Add to dashboard Cite

Large-scale websites are predominantly built as a service-oriented architecture. Here, services are specialized for a certain task, run on multiple machines, and communicate with each other to serve a user's request. An anomalous change in a metric of one service can propagate to other services during this communication, resulting in overall degradation of the request. As any such degradation is revenue impacting, maintaining correct functionality is of paramount concern: it is important to find the root cause of any anomaly as quickly as possible. This is challenging because there are numerous metrics or sensors for a given service, and a modern website is usually composed of hundreds of services running on thousands of machines in multiple data centers. This paper introduces MonitorRank, an algorithm that can reduce the time, domain knowledge, and human effort required to find the root causes of anomalies in such service-oriented architectures. In the event of an anomaly, MonitorRank provides a ranked order list of possible root causes for monitoring teams to investigate. MonitorRank uses the historical and current time-series metrics of each sensor as its input, along with the call graph generated between sensors to build an unsupervised model for ranking. Experiments on real production outage data from LinkedIn, one of the largest online social networks, shows a 26% to 51% improvement in mean average precision in finding root causes compared to baseline and current state-of-the-art methods.

show abstract

VScope: Middleware for Troubleshooting Time-Sensitive Data Center Applications

Cited by 24 publications

References 21 publications

Improving MapReduce performance in heterogeneous environments with adaptive task tuning

Improving MapReduce performance in heterogeneous environments with adaptive task tuning

Combining user interaction, speculative query execution and sampling in the DICE system

Root cause detection in a service-oriented architecture

Contact Info

Product

Resources

About