Multi-Phase Task-Based HPC Applications: Quickly Learning how to Run Fast

Abstract

Parallel applications performance strongly depends on the number of resources. Although adding new nodes usually reduces execution time, excessive amounts are often detrimental as they incur substantial communication overhead, which is difficult to anticipate. Characteristics like network contention, data distribution methods, synchronizations, and how communications and computations overlap generally impact the performance. Finding the correct number of resources can thus be particularly tricky for multi-phase applications as each phase may have very different needs, and the popularization of hybrid (CPU+GPU) machines and heterogeneous partitions makes it even more difficult. In this paper, we study and propose, in the context of a task-based GeoStatistic application, strategies for the application to actively learn and adapt to the best set of heterogeneous nodes it has access to. We propose strategies that use the Gaussian Process method with trends, bound mechanisms for reducing the search space, and heterogeneous behavior modeling. We compare these methods with traditional exploration strategies in 16 different machines scenarios. In the end, the proposed strategies are able to gain up to ≈51% compared to the standard case of using all the nodes while having low overhead.

3D Visualization of Asynchronous Many-Task Scheduling Algorithm

“…Figure 11 -depicts three iterations of ExaGeoStat package where the x -axis is the time, and the y-axis has the aggregated resource type utilization per node. The different colors correspond to different phases: the yellow ones are the generation, while the green ones are the tasks the factorization, a small number of tasks in gray correspond to the other three phases [15].…”

Utilização de aprendizado por reforço para adaptar a quantidade de recursos e acelerar aplicações

“…Embora o paradigma baseado em tarefas atenue a sobrecarga de comunicação, efeitos imprevisíveis (contenção, sincronizações) continuam a ser possíveis e particularmente difíceis de modelar a priori, especialmente quando se explora conjuntos heterogêneos de nós [Nesi et al 2022]. Como a quantidade de nós é diretamente relacionada ao desempenho e a estes efeitos, métodos que aprendem e se adaptam são desejáveis.…”

The Second Competition on Spatial Statistics for Large Datasets

“…The MLOE/MMOM tools are used to assess the loss of the prediction efficiency by using the approximated or misspecified covariance models (Hong et al, 2021). The Fisher information matrix provides information about the importance of a single observation in estimating unknown statistical parameters, i.e., uncertainty Intel Processor Intel Haswell (Abdulah et al, 2018a(Abdulah et al, ,b, 2019, Intel Broadwell (Abdulah et al, 2018a,b), Intel KNL (Abdulah et al, 2018a,b), Intel Skylake (Abdulah et al, 2018b(Abdulah et al, ,2019Salvaña et al, 2022a), and Intel Cascade Lake (Mondal et al, 2022;Salvaña et al, 2022a) AMD Processor AMD EPYC (Salvaña et al, 2022a), and AMD Milan (Mondal et al,2022) ARM Processor ThunderX2 ARM (Salvaña et al,2022a) NVIDIA GPUs NVIDIA K80s (Abdulah et al, 2018a(Abdulah et al, , 2019, NVIDIA P100 (Abdulah et al,2019), and NVIDIA V100 (Abdulah et al, 2019;Salvaña et al, 2022a) Supercomputers KAUST Shaheen-II CrayXC40 (Abdulah et al, 2018a(Abdulah et al, ,b, 2019(Abdulah et al, , 2021Mondal et al,2022;Salvaña et al,2022a,b), HLRS HPE APOLLO (HAWK) , ORNL Summit (Abdulah et al, 2021), INRIA Grid5000 (Nesi et al,2021(Nesi et al, , 2022, and LNCC Santos Dumont supercomputer (SD) (Nesi et al,2022), and Riken Fugaku Fujitsu A64FX (Cao et al,2022) Distributed-Memory Systems…”