Robust clustering analysis for the management of self-monitoring distributed systems

Abstract

We present a decentralized algorithm for online clustering analysis used for anomaly detection in selfmonitoring distributed systems. In particular, we demonstrate the monitoring of a network of printing devices that can perform the analysis without the use of external computing resources (i.e. in-network analysis). We also show how to ensure the robustness of the algorithm, in terms of anomaly detection accuracy, in the face of failures of the network infrastructure on which the algorithm runs. Further, we evaluate the tradeoff in terms of overhead necessary for ensuring this robustness and present a method to reduce this overhead while maintaining the detection accuracy of the algorithm.

Synthesis of application-level utility functions for autonomic self-assessment

“…Its focus is upon the non-analytic self-assessment of a single application, or even a single recognizable service or task executed by that application. Some other nonanalytic approaches exist, based, e.g., on clustering [9] [22]. They are able to distinguish between anomalous and common behavior of an application; however, they cannot quantitatively attribute application states that correspond to either anomalies or normal operation.…”

“…Consequently, it might be unfeasible to synthesize an equation that is valid outside of the main operation areas, and which covers and accurately attributes utility to anomalous data points. On the other hand, we could identify and tell apart different operation areas by analyzing in-the-field data with clustering techniques -as shown for instance by Quiroz et al [22] -and possibly characterize those operation areas in terms of different application features being exercised. That would easily lead to the definition of distinct utility functions for each of them.…”

Autonomic management of distributed systems using online clustering

“…The exchange of job requests is supported by a robust content-based messaging substrate [1], [2], which also enables the partitioning of the information space into regions by implementing a dynamic mapping of points to processing nodes. The messaging substrate is responsible for getting the information used by the clustering analysis to the distributed processing nodes in a scalable fashion.…”

In-situ feature-based objects tracking for data-intensive scientific and enterprise analytics workflows

“…Central to this framework is the scalable decentralized and online clustering (DOC) [6,7] and cluster tracking algorithm, which executes in-situ, i.e., on different cores, and in parallel with the simulation processes, and accesses simulation data directly and asynchronously (to the extent possible) via on-chip shared memory. The framework also provides programming support for composing in-situ "simulation plus analytics" workflows.…”