Lustre Lockahead: Early experience and performance using optimized locking

Abstract

SummaryRecent Cray-authored Lustre modifications known as Lustre Lockahead show significantly improved write performance for collective, shared-file I/O workloads. Initial tests show write performance improvements of more than 200% for small transfer sizes and over 100% for larger transfer sizes compared to traditional Lustre locking. Standard Lustre shared-file locking mechanisms limit scaling of shared-file I/O performance on modern high-performance Lustre servers.The new Lockahead feature provides a mechanism for applications (or libraries) with knowledge of their I/O patterns to overcome this limitation by explicitly requesting locks. MPI-IO is able to use this feature to dramatically improve shared-file collective I/O performance, achieving more than 80% of file per process performance. This paper discusses our early experience using Lockahead with applications. We also present application and synthetic performance results and discuss performance considerations for applications that benefit from Lockahead. INTRODUCTIONPOSIX I/O file access behavior is usually categorized as file-per-process or single-shared-file. Historically, file-per-process access has provided higher throughput than single-shared-file access due to required file system overhead ensuring consistency during shared write accesses. However, optimal I/O throughput is never the only, and rarely the primary, consideration when writing or using an application. Despite the performance downside, shared files are widely used for a variety of data management and ease of use reasons. To avoid long I/O time caused by shared-file performance, an application may reduce data output, checkpoint frequency, or number of jobs.Shared-file performance characterization is largely specific to each file system implementation and I/O library. This work focuses on modifications to the Lustre file system for applications using collective MPI-IO operations. A design and implementation in Lustre and Cray MPI libraries to improve shared-file write performance by introducing a new Lustre locking scheme is described in this paper. Given MPI-IO allows requesting different file system lock modes via environment variables, this work provides a path for applications to improve performance without application code modifications. Code changes within Lustre and MPICH/ROMIO to support Lustre Lockahead have been contributed back to the upstream communities. 1,2 This paper first describes application I/O behavior and standard libraries used for shared-file access. The current Lustre shared-file locking implementation is also described to motivate the need for an improved locking mechanism. Second, the implementation of Lustre Lockahead for collective MPI-IO operations is described for both the Lustre file system and the collective MPI-IO library. Next, comparative I/O performance of current file-per-process, independent and collective shared-file I/O is presented to evaluate the benefit of this new locking method for Lustre. Finally, we describe early experience using Lustre Lock...

Foreword to the special issue of the Cray User Group (CUG 2017)

“…The best paper award at CUG 2017 went to a paper titled “Lustre Lockahead: Early Experience and Performance using Optimized Locking.” The focus of this paper on optimizing performance of shared file writing on Lustre underscores the continued emphasis on I/O optimization that we have seen in many past CUG conferences. Several papers() in this special issue focus on I/O optimization for large scale Cray systems.…”

Lustre I/O performance investigations on Hazel Hen: experiments and heuristics

“…The following results section will contain variation of several I/O parameters: The number of employed processes, the local domain size per process (block size), the stripe count, the stripe size and the aggregator node multiplier in the novel Lustre Lock-ahead (LLA) locking mechanism [34]. The ranges of the parameters can be found in Table 1, but note that the space spanned by these was subsampled based on prior results.…”

“…As we have already seen, the performance for 8 nodes levels off around the maximum number of OSTs available, most likely due to lock contention between aggregators. In order to get higher throughput from the OSTs, CRAY provides a novel locking mechanism called Lustre Lock-ahead (LLA) on ws9, which allows multiple aggregating I/O processes to write concurrently to the same OST [34].…”