The NetLogger methodology for high performance distributed systems performance analysis

Abstract

We describe a methodology that enables the real-time diagnosis of performance problems in complex high-performance distributed systems. The IntroductionDevelopers of high-speed network-based distributed systems often observe performance problems such as unexpectedly low network throughput or high latency. The reasons for the poor performance can be manifold and are frequently not obvious. It is often difficult to track down performance problems because of the complex interaction between the many distributed system components, and the fact that performance problems in one place may be most apparent somewhere else. Bottlenecks can occur in any of the components along the paths through which the data flow: the applications, the operating systems, the device drivers, the network adapters on any of the sending or receiving hosts, and/or in network components such as switches and routers. Sometimes bottlenecks involve interactions among several components or the interplay of protocol parameters at different points in the system, and sometimes, of course, they are due to unrelated network activity impacting the operation of the distributed system. While post-hoc diagnosis of performance problems is valuable for systemic problems, for operational problems users will have already suffered through a period of degraded performance. The ability to recognize operational problems would enable elements of the distributed system to use this information to adapt to operational conditions, minimizing the impact on users.We have developed a methodology, known as NetLogger, for monitoring, under realistic operating conditions, the behavior of all the elements of the application-to-application communication path in order to determine exactly what is happening within a complex system. Distributed application components, as well as some operating system components, are modified to perform precision timestamping and logging of "interesting" events, at every critical point in the distributed system. The events are correlated with the system's behavior in order to characterize the performance of all aspects of the system and network in detail during actual operation. The monitoring is designed to facilitate identification of bottlenecks, performance tuning, and network performance research. It also allows accurate measurement of throughput and latency characteristics for distributed application codes.Software agents collect and filter event-based performance information, turn on or off various monitoring options to adapt the monitoring to the current system state, and manage the large amounts of log data that are generated.The goal of this performance characterization work is to produce high-speed components that can be used as building blocks for high-performance applications, rather than having to "tune" the applications top-to-bottom as is all too common today. This method can also provide an information source for applications that can adapt to component congestion problems.NetLogger has demonstrated its usefulness with th...

Enabling privacy through transparency

“…generating log files that incorporate dependencies between entities, the start and stop times of activities, and the inputs/outputs of activities, has been a topic of interest in many programming languages research [16]. There are many infrastructures, specifically in web service adaptors, that can be repurposed for collecting provenance [17].…”

Online Critical Path Profiling for Parallel Applications

“…Of existing trace-based monitoring systems, only NetLogger [28] allows online monitoring by transmitting collected events to a central collection node for runtime analysis. This allows it to obtain a global view of system behavior.…”

“…Trace-based profiling approaches [1,3,9,11,15,20,21,22,24,25,28,31] can provide detailed information of the whole system, including cross-node performance information such as profiling of the distributed critical path of the entire application.…”

Multi-dimensional Knowledge Integration for Efficient Incident Management in a Services Cloud

“…Monitoring systems are widely deployed to detect system/application events and collect performance data [15]. Some monitoring systems today are integrated with service request management systems to automatically open incident tickets upon detection of certain events [11].…”