Real-Time Statistical Clustering for Event Trace Reduction

Abstract

Event tracing provides the detailed data needed to understand the dynamics of interactions among application resource demands and system responses. However, capturing the large volume of dynamic performance data inherent in detailed tracing can perturb program execution and stress secondary storage systems. Moreover, it can overwhelm a user or performance analyst with potentially irrelevant data. Using the Pablo performance environment's support for real-time data analysis, we show that dynamic statistical data clustering can dramatically reduce the volume of captured performance data by identifying and recording event traces only from representative processors. In turn, this makes possible low overhead, interactive visualization, and performance tuning.

Scalable methods for monitoring and detecting behavioral equivalence classes in scientific codes

“…Nikolayev et al [14] have used statistical clustering to reduce data overhead in large applications by sampling only representatives from clusters detected at runtime. The approach is entirely online: clusters are generated on-the-fly based on data observed at runtime.…”

Evaluating similarity-based trace reduction techniques for scalable performance analysis

“…Aguilera et al [2], Nickolayev et al [25], and Lee et al [23] apply statistical clustering to traces and select a representative trace for each cluster of processes. Nickolayev and Lee use the Euclidean distance for clustering, while Aguilera uses a metric based on the amount of communication between two processes.…”

Trace reduction for performance improvement assessment of message passing parallel programs

“…Next, information aggregation strategies [10]- [12] probably fail to keep information (A) in traces. Therefore, in many cases, PerWiz is unable to assess the improvability of programs by using their traces.…”

“…Therefore, in many cases, PerWiz is unable to assess the improvability of programs by using their traces. The first method [10], which focuses on the similarly between process behaviors, stops recording events occurred on non-representative processes. Therefore, if the CP consists of such unrecorded events, PerWiz fails to compute the CP.…”

“…For example, Nickolayev et al [10] propose a runtime method that periodically evaluates the similarity of process behaviors in order to generate traces only for some processes that are representatives of the remaining similar processes. Pablo [11] focuses on the frequency of events in order to select the appropriate level of details for trace generation.…”