Comparing high resolution timestamps in computer clusters

Abstract

Gaining a deep understanding of the detailed execution behavior and performance of complex software systems requires accurate tracing and analysis tools. Tracing tools such as the Linux Trace Toolkit (LTT) provide a detailed log of both system and user level events which may later be accessed and visualized. The information logged for each event includes the event type, a timestamp, and event type specific information. Most computer central processing units (CPU) now come with a cycle counter, thus enabling fast access, very high resolution (cycle level), timestamp values for events.Problems arise when tracing events from computer clusters containing numerous CPUs with independent clocks. Even CPUs in the same multiprocessor computer may not be fully synchronized, and in some cases run at different clock frequencies. Visualization tools, when presenting a timeline of events logged on different CPUs must therefore translate the individual timestamps to a common time reference, thus taking into account the clock delay for each CPU. Several experiments were conducted in order to characterize the CPU cycle counter accuracy and measure the synchronization accuracy achievable using conventional computer networks. The resulting data is of immediate use for all applications concerned with computer clocks or network synchronization accuracy. It will be used in the forthcoming version of the Linux Trace Toolkit to select the most appropriate mechanisms to compute a common time reference for events coming from different CPUs.

Internal Clock Drift Estimation in Computer Clusters

“…For our experiment, the lowest-level interrupt handler in Linux kernel version 2.4.26 was instrumented, as shown in Algorithm 1, to sample the cycle counter as soon as the PPS signal interrupt is notified [14,15]. Thereafter, the difference between two successive readings is calculated, yielding the number of clock cycles per second, or clock frequency, for the CPU.…”

Instrumentation-based tool for latency measurements

“…Accuracy of the system clock is dependent on the quartz crystal it is made of, and thus its accuracy depends on temperature. Accuracy and drifting of the clock over time were found to be modest in modern microprocessor architectures [3]. However, the resolution is constrained by frequency of the crystal to around 1µs [25].…”

“…Trimble Lassen iQ GPS manual reports up to 50 ns accuracy for the rising edge of PPS, which is synchronized with UTC time. Delay for the signal propagation from GPS device via the adapter to the laptop should be constant [3]. However, the internal delay of the interrupt handler in PC is variable and dependent on the processing load in non-real-time operating systems.…”

“…Accuracy of high resolution timestamps on Linux platform was measured by the authors of [3] for the development of the Linux Trace Toolkit. It enables the tracing of program execution flow without instrumentation based on timestamps embedded to the kernel [4].…”

Fail-Aware Publish/Subscribe

“…For modern processors (clock frequencies ≈ 3GHz) this allows for over 190 years of operation without overflow. The authors of [14] have indicated that TSCs are stable and are well suited for duration measurements. In order to verify the applicability of TSCs in the PlanetLab environment, we have conducted our own measurements (see Section 9) which allowed us to determine that the PlanetLab nodes' TSC drift rates are sufficiently stable to be used with F-A P/S.…”