SPIDAL Java: High Performance Data Analytics with Java and MPI on Large Multicore HPC Clusters

Abstract

Within the last few years, there have been significant contributions to Java-based big data frameworks and libraries such as Apache Hadoop, Spark, and Storm. While these systems are rich in interoperability and features, developing high performance big data analytic applications is challenging. Also, the study of performance characteristics and high performance optimizations is lacking in the literature for these applications. By contrast, these features are well documented in the High Performance Computing (HPC) domain and some of the techniques have potential performance benefits in the big data domain as well. This paper presents the implementation of a high performance big data analytics library-SPIDAL Java-with a comprehensive discussion on five performance challenges, solutions, and speedup results. SPIDAL Java captures a class of global machine learning applications with significant computation and communication that can serve as a yardstick in studying performance bottlenecks with Java big data analytics. The five challenges present here are the cost of intra-node messaging, inefficient cache utilization, performance costs with threads, overhead of garbage collection, and the costs of heap allocated objects. SPIDAL Java presents its solutions to these and demonstrates significant performance gains and scalability when running on up to 3072 cores in one of the latest Intel Haswell-based multicore clusters.

Twister2: Design of a big data toolkit

“…There are synergies between HPC and big data systems, and authors 29,30 among others 31 have expressed the need to enhance these systems by taking ideas from each other. In previous work 32,33 we have identified the general implications of threads and processes, cache, memory management in NUMA 34 , as well as multi-core settings for machine learning algorithms with MPI. DataMPI 35 uses MPI to build Hadoop-like systems while 36 uses MPI communications in Spark for better performance.…”

“…An MPI programmer has to consider low-level details such as I/O, memory hierarchy and efficient execution of threads to write a parallel application that scales to large numbers of nodes. With the increasing availability of multi-core and many-core systems, the burden on the programmer to get the best available performance has increased dramatically 32,33 . Asynchronous many task systems are such as HPX 42 , Legion 41 , DAGuE 38 are developed to hide some of these complexities.…”