PCJ – Java Library for Highly Scalable HPC and Big Data Processing

Nowicki, Marek; Górski, Łukasz; Bała, Piotr

doi:10.1109/hpcs.2018.00017

Cited by 7 publications

(10 citation statements)

References 3 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…As a result, a higher performance is achieved; unfortunately, it is still lower than for other Java-based solutions. In particular, it has been demonstrated that the performance of applications built using the PCJ library or APGAS (a branch of IBM's X10 language project) is higher than for Hadoop/Spark implementations [10,11]. For the calculation-intensive and communication-intensive HPC workloads, PCJ is several up to hundreds of times faster [12].…”

Section: Map-reducementioning

confidence: 99%

Scalable computing in Java with PCJ Library. Improved collective operations

Nowicki¹,

Górski²,

Bała³

2021

Proceedings of International Symposium on Grids &Amp; Clouds 2021 — PoS(ISGC2021)

Self Cite

View full text Add to dashboard Cite

Machine learning and Big Data workloads are becoming as important as traditional HPC ones. AI and Big Data users tend to use new programming languages such as Python, Julia, or Java, while the HPC community is still dominated by C/C++ or Fortran. Hence, there is a need for new programming libraries and languages that will integrate different applications and allow them to run on large computer infrastructure. Since modest computers are multinode and multicore, parallel execution is an additional challenge here. For that purpose, we have developed the PCJ library, which introduces parallel programming capabilities to Java using the Partitioned Global Address Space model. It does not modify language nor running environment (JVM). The PCJ library allows for easy development of parallel code and runs it on laptops, workstations, supercomputers, and the cloud. This paper presents an overview of the PCJ library and its usage in parallelizing selected workloads, including HPC, AI, and Big Data. The performance and scalability are presented. We present recent addition to the PCJ library, which are collective operations. The collective operations significantly reduce the number of lines of code to write, ensuring good performance.

show abstract

Section: Map-reducementioning

confidence: 99%

Scalable computing in Java with PCJ Library. Improved collective operations

Nowicki¹,

Górski²,

Bała³

2021

Proceedings of International Symposium on Grids &Amp; Clouds 2021 — PoS(ISGC2021)

Self Cite

View full text Add to dashboard Cite

show abstract

“…PCJ 2,33,34 is an open‐source Java library that does not require any language extensions nor special compiler. The user has to download the single jar file from the library homepage or use build automation tool with dependency resolvers like Maven or Gradle as the library is also available from Maven Central Repository ( group‐id : pl.edu.icm.pcj, artifact‐id : pcj).…”

Section: The Pcj Librarymentioning

confidence: 99%

“…Network communication is performed using Java New IO classes (i.e., java.nio.*). The details of the algorithms used to implement PCJ communication are described in our previous publications 2,3 …”

Section: The Pcj Librarymentioning

confidence: 99%

“…In previous works, we have shown that the PCJ library allows for easy and feasible development of computational applications as well as big data and AI processing running on supercomputers or clusters 3 . The performance comparison with the C/MPI based codes has been presented in previous papers 4,5 . The extensive comparison with Java‐based solutions including APGAS (Java implementation of X10 language) has been also performed 3,6 …”

Section: Introductionmentioning

confidence: 99%

“…Usage of well‐established industrial language opens the field for easy integration of both workload types. The PCJ library (parallel computing in Java) 2 fully complies with core Java standards, therefore, the programmer does not have to use additional libraries, which are not part of the standard Java distribution. Thus, the user is free from the burden of installing (properly versioned) dependencies, as the library is a single self‐contained jar file, which can be easily dropped into the classpath .…”

Section: Introductionmentioning

confidence: 99%

See 2 more Smart Citations

Performance evaluation of Java/PCJ implementation of parallel algorithms on the cloud (extended version)

Nowicki

Górski

Bała

2021

Concurrency and Computation

Self Cite

View full text Add to dashboard Cite

Large-scale computing and data processing with cloud resources is gaining popularity.However, the usage of the cloud differs from traditional high-performance computing (HPC) systems and both algorithms and codes have to be adjusted. This work is often time-consuming and performance is not guaranteed. To address this problem we have developed the PCJ library (parallel computing in Java), a novel tool for scalable HPC and big data processing in Java. In this article, we present a performance evaluation of parallel applications implemented in Java using the PCJ library. The performance evaluation is based on the examples of highly scalable applications of different characteristics focusing on CPU, communication or I/O. They run on the traditional HPC system and Amazon web services Cloud as well as Linaro Developer Cloud. For the clouds, we have used Intel x86 and ARM processors for running Java codes without changing any line of the program code and without the need for time-consuming recompilation. Presented applications have been parallelized using the partitioned global address space programming model and its realization in the PCJ library. Our results prove that the PCJ library, due to its performance and ability to create simple portable code, has great promise to be successful for the parallelization of various applications and run them on the cloud with a performance close to HPC systems.

show abstract

A multithreaded CUDA and OpenMP based power‐aware programming framework for multi‐node GPU systems

Czarnul

2023

Concurrency and Computation

View full text Add to dashboard Cite

SummaryIn the article, we have proposed a framework that allows programming a parallel application for a multi‐node system, with one or more graphical processing units (GPUs) per node, using an OpenMP+extended CUDA API. OpenMP is used for launching threads responsible for management of particular GPUs and extended CUDA calls allow to transfer data and launch kernels on local and remote GPUs. The framework hides inter‐node MPI communication from the programmer. For optimization, the implementation takes advantage of the MPI_THREAD_MULTIPLE mode allowing: multiple threads handling distinct GPUs as well as overlapping communication and computations transparently using multiple CUDA streams. The solution allows data parallelization across available GPUs in order to minimize execution time and supports a power‐aware mode in which GPUs are automatically selected for computations using a greedy approach in order not to exceed an imposed power limit. We have implemented and benchmarked three parallel applications including: finding the largest divisors; verification of the Collatz conjecture; finding patterns in vectors. These were tested on three various systems: a GPU cluster with 16 nodes, each with NVIDIA GTX 1060 GPU; a powerful 2‐node system—one node with 8 NVIDIA Quadro RTX 6000 GPUs, the second with 4 NVIDIA Quadro RTX 5000 GPUs; a heterogeneous environment with one node with 2 NVIDIA RTX 2080 and 2 nodes with NVIDIA GTX 1060 GPUs. We demonstrated effectiveness of the framework through execution times versus power caps within ranges of 100–1400 W, 250–3000 W, and 125–600 W for these systems respectively as well as gains from using two versus one CUDA streams per GPU. Finally, we have shown that for the testbed applications the solution allows to obtain high speed‐ups between 89.3% and 97.4% of the theoretically assessed ideal ones, for 16 nodes and 2 CUDA streams, demonstrating very good parallel efficiency.

show abstract

PCJ – Java Library for Highly Scalable HPC and Big Data Processing

Cited by 7 publications

References 3 publications

Scalable computing in Java with PCJ Library. Improved collective operations

Scalable computing in Java with PCJ Library. Improved collective operations

Performance evaluation of Java/PCJ implementation of parallel algorithms on the cloud (extended version)

A multithreaded CUDA and OpenMP based power‐aware programming framework for multi‐node GPU systems

Contact Info

Product

Resources

About