Acceleration of a Full-Scale Industrial CFD Application with OP2

Abstract

Hydra is a full-scale industrial CFD application used for the design of turbomachinery at Rolls Royce plc. It consists of over 300 parallel loops with a code base exceeding 50K lines and is capable of performing complex simulations over highly detailed unstructured mesh geometries. Unlike simpler structured-mesh applications, which feature high speed-ups when accelerated by modern processor architectures, such as multi-core and many-core processor systems, Hydra presents major challenges in data organization and movement that need to be overcome for continued high performance on emerging platforms. We present research in achieving this goal through the OP2 domain-specific high-level framework. OP2 targets the domain of unstructured mesh problems and follows the design of an active library using source-to-source translation and compilation to generate multiple parallel implementations from a single high-level application source for execution on a range of back-end hardware platforms. We chart the conversion of Hydra from its original hand-tuned production version to one that utilizes OP2, and map out the key difficulties encountered in the process. To our knowledge this research presents the first application of such a high-level framework to a full scale production code. Specifically we show (1) how different parallel implementations can be achieved with an active library framework, even for a highly complicated industrial application such as Hydra, and (2) how different optimizations targeting contrasting parallel architectures can be applied to the whole application, seamlessly, reducing developer effort and increasing code longevity. Performance results demonstrate that not only the same runtime performance as that of the hand-tuned original production code could be achieved, but it can be significantly improved on conventional processor systems. Additionally, we achieve further acceleration by exploiting many-core parallelism, particularly on GPU systems. Our results provide evidence of how high-level frameworks such as OP2 enable portability across a wide range of contrasting platforms and their significant utility in achieving near-optimal performance without the intervention of the application programmer.

Optimizations of Unstructured Aerodynamics Computations for Many-core Architectures

“…Later in [13], [14], [16], crucial efforts of evaluating the thread-level performance potentials of PETSc-FUN3D on wide spectrum of architectures are presented. On the other hand, SU2 code of Stanford [59] and OP2 code of Oxford [60] are considered to be the state-of-the-practice unstructured CFD research codes, which both have recently been ported into many emerging HPC architectures [61], [62].…”

Achieving Performance Portability for a Heat Conduction Solver Mini-Application on Modern Multi-core Systems

“…One similarity in all definitions was the intuition that a performance portable code should be able to run on a variety of machines. There have been many different approaches to solve this, including compiler directives such as OpenACC [24] and OpenMP, languages designed for performance portability such as Chapel [25] and PetaBricks [26], execution models such as EARTH [27], and using embedded domain specific languages such as OPS [6] and OP2 [28]. Template libraries have also been used to add performance portability to an application, examples of which include Kokkos [7] and RAJA [8].…”

Performance Analysis of a High-Level Abstractions-Based Hydrocode on Future Computing Systems

“…Research published as a result of this work includes a number of performance analysis studies on standard CFD benchmark applications [23] as well as a full industrial-scale application from the production work-load at Rolls-Royce plc. [28]. OPS (Oxford Parallel Library for Structured-mesh solvers) follows much of the design of OP2, but targets the domain of multi-block structured applications.…”

“…It is then the task of a lower implementation level to apply automated techniques for translating the specification into different implementations for different hardware and software platforms. The use of such a development strategy has previously been shown to have significant benefits both for developer productivity and gaining near-optimal performance [28,14]. However, currently these still remain as experimental research projects and have not yet been adopted by a wider HPC community.…”