Simplifying the Parallelization of Highly Concurrent Workloads

Berlich, Ruediger; Gabriel, Sven; García, Ariel Oscar

doi:10.22323/1.239.0026

Cited by 3 publications

(6 citation statements)

References 0 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…If the system had been implemented without asynchronous requests, as the consumers that have existed so far in Geneva, scalability would be limited by Amdahl's law [35] [32,5]. According to Amdahl, the minimum execution time achievable by parallel evaluation of candidate solutions is the sum t min = t seq + t par of the time t seq spent sequentially on server-side computation (i.e., primarily (de)serialization) and the time t par required for parallel execution on clients.…”

Section: Asynchronous Requestsmentioning

confidence: 99%

“…The reason is that asynchronous requests can effectively circumvent Amdahl's law by allowing sequential computations on the server to overlap in time with computations on the clients, so that t min ≈ t par holds. This method had also been briefly proposed by Dr. Berlich and is now implemented as part of the MPI Consumer [32,6].…”

Section: Asynchronous Requestsmentioning

confidence: 99%

“…In this section, we give instructions for using the MPI-Consumer with the Geneva program library. For general usage hints for Geneva independent of the MPI-Consumer, refer to the Geneva manual [11], the paper written by Dr. Berlich in [32] and the examples in the Github repository [1].…”

Section: Appendix a Mpi Consumer User Guidementioning

confidence: 99%

“…For historical reasons, the sublibrary has the same name as the program library collection[32,2].5 also called fitness function or cost function…”

mentioning

confidence: 99%

See 3 more Smart Citations

Parametric Optimization on HPC Clusters with Geneva

Weßner

Berlich²,

Schwarz

et al. 2023

Preprint

View full text Add to dashboard Cite

Many challenges of today's science are parametric optimization problems that are extremely complex and computationally intensive to calculate. At the same time, the hardware for high-performance computing is becoming increasingly powerful. Geneva is a framework for parallel optimization of large-scale problems with highly nonlinear quality surfaces in grid and cloud environments. To harness the immense computing power of high-performance computing clusters, we have developed a new networking component for Geneva, the so-called MPI Consumer, which makes Geneva suitable for HPC. Geneva is most prominent for its evolutionary algorithm, which requires repeatedly evaluating a user-defined cost function. The MPI Consumer parallelizes the computation of the candidate solutions' cost functions by sending them to remote cluster nodes. By using an advanced multithreading mechanism on the master node and by using asynchronous requests on the worker nodes, the MPI Consumer is highly scalable. Additionally, it provides fault tolerance, which is usually not the case for MPI programs but becomes increasingly important for HPC. Moreover, the MPI Consumer provides a framework for the intuitive implementation of fine-grained parallelization of the cost function. Since the MPI Consumer conforms to the standard paradigm of HPC programs, it vastly improves Geneva's user-friendliness on HPC clusters. This article gives insight into Geneva's general system architecture and the system design of the MPI Consumer as well as the underlying concepts. Geneva, including the novel MPI Consumer, is publicly available as an open source project on GitHub and is currently used for fundamental physics research at GSI in Darmstadt, Germany.

show abstract

Section: Asynchronous Requestsmentioning

confidence: 99%

Section: Asynchronous Requestsmentioning

confidence: 99%

Section: Appendix a Mpi Consumer User Guidementioning

confidence: 99%

“…For historical reasons, the sublibrary has the same name as the program library collection[32,2].5 also called fitness function or cost function…”

mentioning

confidence: 99%

See 2 more Smart Citations

Parametric Optimization on HPC Clusters with Geneva

Weßner

Berlich²,

Schwarz

et al. 2023

Preprint

View full text Add to dashboard Cite

show abstract

“…In this section, we give instructions for using the MPI-Consumer with the Geneva program library. For general usage hints for Geneva independent of the MPI-Consumer, refer to the Geneva manual [10], the paper written by Dr. Berlich in [31] and the examples in the Github repository [38].…”

Section: Appendix: Mpi Consumer User Guidementioning

confidence: 99%

Parametric Optimization on HPC Clusters with Geneva

Weßner

Berlich

Schwarz

et al. 2023

Comput Softw Big Sci

View full text Add to dashboard Cite

Many challenges of today’s science are parametric optimization problems that are extremely complex and computationally intensive to calculate. At the same time, the hardware for high-performance computing is becoming increasingly powerful. Geneva is a framework for parallel optimization of large-scale problems with highly nonlinear quality surfaces in grid and cloud environments. To harness the immense computing power of high-performance computing clusters, we have developed a new networking component for Geneva—the so-called MPI Consumer—which makes Geneva suitable for HPC. Geneva is most prominent for its evolutionary algorithm, which requires repeatedly evaluating a user-defined cost function. The MPI Consumer parallelizes the computation of the candidate solutions’ cost functions by sending them to remote cluster nodes. By using an advanced multithreading mechanism on the master node and by using asynchronous requests on the worker nodes, the MPI Consumer is highly scalable. Additionally, it provides fault tolerance, which is usually not the case for MPI programs but becomes increasingly important for HPC. Moreover, the MPI Consumer provides a framework for the intuitive implementation of fine-grained parallelization of the cost function. Since the MPI Consumer conforms to the standard paradigm of HPC programs, it vastly improves Geneva’s user-friendliness on HPC clusters. This article gives insight into Geneva’s general system architecture and the system design of the MPI Consumer as well as the underlying concepts. Geneva—including the novel MPI Consumer—is publicly available as an open source project on GitHub (https://github.com/gemfony/geneva) and is currently used for fundamental physics research at GSI in Darmstadt, Germany.

show abstract

Speeding up Science Through Parametric Optimization on HPC Clusters

Wessner,

Berlich,

Schwarz

et al. 2023

Proceedings of International Symposium on Grids &Amp; Clouds (ISGC) 2023 in Conjunction With HEPiX Spring 2023 Workshop — PoS(I

View full text Add to dashboard Cite

Science is constantly encountering parametric optimization problems whose computer-aided solutions require enormous resources. At the same time, there is a trend towards the development of increasingly powerful computer clusters. Geneva is currently one of the best available frameworks for distributed optimization of large-scale problems with highly nonlinear quality surfaces. It is an excellent tool to be used in wide-area networks such as Grids and Clouds. However, it was not user-friendly for scheduling on high-performance computing clusters and supercomputers. Another issue was that it only provided a framework for parallelizing workloads at the population level of optimization algorithms, but did not support distributed parallelization of the cost function itself. For this reason, a new software component for network communication -called MPI-Consumer -has been developed and published as open-source software. As a complement to our previous paper, which explained the MPI Consumer's system design, this article provides an extensive performance evaluation. Comparing the MPI Consumer with two earlier in Geneva developed network technologies shows that the MPI Consumer is an overall improvement in terms of performance. Furthermore, we analyze the impact of system components of the MPI Consumer by testing different configurations. It is observed that asynchronous client requests speed up the time to solution by up to 20%. Furthermore, the multithreading design proves to be very scalable, allowing for a significant speed-up even if on extremely high loads. Tests on GSI's Green IT Cube HPC cluster show that our measurements realistically reflect the expected behavior in a production environment.

show abstract

Simplifying the Parallelization of Highly Concurrent Workloads

Cited by 3 publications

References 0 publications

Parametric Optimization on HPC Clusters with Geneva

Parametric Optimization on HPC Clusters with Geneva

Parametric Optimization on HPC Clusters with Geneva

Speeding up Science Through Parametric Optimization on HPC Clusters

Contact Info

Product

Resources

About