Replay-Based Synchronization of Timestamps in Event Traces of Massively Parallel Applications

Abstract

Event traces are helpful in understanding the performance behavior of message-passing applications since they allow in-depth analyses of communication and synchronization patterns. However, the absence of synchronized hardware clocks may render the analysis ineffective because inaccurate relative event timings can misrepresent the logical event order and lead to errors when quantifying the impact of certain behaviors. Although linear offset interpolation can restore consistency to some degree, inaccuracies and time-dependent drifts may still disarrange the original succession of events -especially during longer runs. In our earlier work, we have presented an algorithm that removes the remaining violations of the logical event order postmortem and, in addition, have outlined the initial design of a parallel version. Here, we complete the parallel design and describe its implementation within the SCALASCA trace-analysis framework. We demonstrate its suitability for large-scale applications running on more than thousand application processes and show how the correction can improve the trace analysis of a real-world application example.

Clock synchronization in high‐end computing environments: a strategy for minimizing clock variance at runtime

“…One technique for establishing a synchronized view of time in a distributed parallel application is to perform the synchronization post‐mortem . This technique is especially well‐suited for event tracing and similar applications where data can be captured along with enough causal and ordering information for the event data to be retroactively corrected.…”

“…If the receiving process's clock is set to an earlier time than the sending process's clock, the messages appear out of order. Techniques exist for addressing this issue in post‐mortem analysis by recording each process's clock skew and correcting for it at log analysis time. In contrast, our technique works by synchronizing the clocks at run time, thus ensuring that the logs contain accurate temporal information as it is captured.…”

Scalable detection of MPI-2 remote memory access inefficiency patterns

“…However, as part of our future work, we plan to address also passive target synchronization. (Becker et al, 2009). The scalability of the parallel replay mechanism has already been demonstrated for up to 294,912 cores (Geimer et al, 2010).…”

An evaluation of the state of time synchronization on leadership class supercomputers

“…With NTP, accuracies are limited to one millisecond in the best case if a time server is available on the local intranet. 30 We believe that improving the resolution of time synchronization within contemporary supercomputers will help to drive system software use cases that leverage tightly synchronized node clocks to improve the performance and efficiency of system software.…”