Design and Prototyping of Container-Enabled Cluster for High Performance Data Analytics

Kwon, JinCheol; Kim, Namgon Lucas; Kang, Moonjoong; WonKim, Jong

doi:10.1109/icoin.2019.8718135

Cited by 9 publications

(2 citation statements)

References 2 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…With the SmartX Intelligence the usage of parallel file system with high-performance within the work node has increase in the performance with the best integration of software for deep learning workloads. [3]…”

Section: Cost-effective Routing For Hpc Networkmentioning

confidence: 99%

Structural Analysis of Hpc’s for Big Data Analytics

S¹,

Koundinya²

2019

IJEAST

View full text Add to dashboard Cite

Data analytics possess challenges in various scenarios, designing a system is one of them. This paper provides an insightful preview of the challenges and possible outcome one needs to consider. The High-Performance Computing system for analysis of complex computer-intensive data needs to be focused on architecture design, network hardware, clustering, I/O systems, Metadata, cost and how this analysis can be used in Machine Learning and Big Data analytics techniques.

show abstract

Section: Cost-effective Routing For Hpc Networkmentioning

confidence: 99%

Structural Analysis of Hpc’s for Big Data Analytics

S¹,

Koundinya²

2019

IJEAST

View full text Add to dashboard Cite

show abstract

“…Kwon et al [33] firstly proposed several hurdles to realize container-enabled clusters for HPDA (high-performance data analytics) workload; Secondly, to address technical constraints around bottlenecked inter-connections for overlay networking and storage access, they designed a containerenabled cluster prototype. Their results show an effectively and seamlessly integration with the hardware and software pieces of HPDA cluster.…”

Section: Introductionmentioning

confidence: 99%

A Task-Resource Mapping Algorithm for Large-Scale Batch-Mode Computational Marine Hydrodynamics Codes on Containerized Private Cloud

Liu

Penesis

et al. 2019

IEEE Access

View full text Add to dashboard Cite

CPU time has long been a remaining problem for large-scale batch mode based scientific computing applications. To address this time-consuming problem, a container-based private cloud was employed, and a novel task-resource mapping algorithm was developed. Firstly, the execution features of typical batch mode codes were extracted and then computing jobs were formulated as a coarseness acyclic DAG. Secondly, to guarantee both job makespan and resource utilization, a novel task-resource mapping algorithm, along with container pre-planning and worst-case-first task placement phases, were developed. Finally, a typical Computational Marine Hydrodynamics software, Rotorysics, with a different scale of input data matrix was used as benchmark software. To manifest the effectiveness of the proposed method, a number of numerical examples were given via CloudSim and a small-medium containerized private cloud platform was adopted with three practical study cases. The computational results show that 1) compared with the traditional HPC workstation computing solution, container-based cloud solution shows significant savings in makespan by more than 6 times. 2) the new method is scalable to address bigger size batch computing problem up to a run matrix 10 8. INDEX TERMS Computational marine hydrodynamics (CMH) codes, containerization, large-scale batch mode computing, private cloud, task-resource mapping algorithm.

show abstract

Workflow Improvement for KubeFlow DL Performance over Cloud-Native SmartX AI Cluster

Hong

Kim

2020

Advances in Internet, Data and Web Technologies

View full text Add to dashboard Cite

Design and Prototyping of Container-Enabled Cluster for High Performance Data Analytics

Cited by 9 publications

References 2 publications

Structural Analysis of Hpc’s for Big Data Analytics

Structural Analysis of Hpc’s for Big Data Analytics

A Task-Resource Mapping Algorithm for Large-Scale Batch-Mode Computational Marine Hydrodynamics Codes on Containerized Private Cloud

Workflow Improvement for KubeFlow DL Performance over Cloud-Native SmartX AI Cluster

Contact Info

Product

Resources

About