Hybrid hierarchy storage system in MilkyWay-2 supercomputer

Xu, Weiya; Lu, Yutong; Li, Qiong; Zhou, Enqiang; Dong, Yong; Zhang, Wei; Wei, Dengping; Zhang, Xiaoming; Chen, Haitao; Xing, Jianying; Yuan, Yuan

doi:10.1007/s11704-014-3499-6

Cited by 43 publications

(18 citation statements)

References 11 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…With the integration of high-performance computing, intelligent computing, and big data analytics, workloads running on supercomputers will be more complex and diverse. As a result, new challenges are put forward in the multiple level hybrid architecture (Xu et al 2014) design of future HPC memory and storage systems.…”

Section: Memory and Storage Systemsmentioning

confidence: 99%

Paving the way for China exascale computing

2019

CCF Trans. HPC

Self Cite

View full text Add to dashboard Cite

Supercomputing technology has been evolving rapidly in an accelerated way. It has made a significant impact on scientific research, technology innovation, economic and social development, and the life of ordinary people. Over the past three decades, China has devoted considerable efforts on the development of supercomputing technologies, and made tremendous and remarkable achievements in this field. China's supercomputing systems now rank among the world's most powerful supercomputers. As Moore's Law approaches its limit, the development of exascale supercomputing systems is facing a series of grand challenges in both technologies and applications. Based on the experiences of China's supercomputing development over the past years, this paper analyzes the major technical challenges on the path towards exascale computing. Additionally, ongoing major R&D activities on next-generation supercomputing in China are introduced, and the possible solutions to achieve exascale computing, including co-design and convergence computing, are discussed.

show abstract

Section: Memory and Storage Systemsmentioning

confidence: 99%

Paving the way for China exascale computing

2019

CCF Trans. HPC

Self Cite

View full text Add to dashboard Cite

show abstract

“…Tianhe-2's 16000 computing nodes do not have a local I/O stack, but instead, all I/O operations are transferred the 256 available intermediate I/O nodes. These nodes are powered with high-speed SSDs, and configurations for each file determine when data is transferred from the I/O nodes to the parallel file system [109].…”

Section: The I/o Forwarding Layermentioning

confidence: 99%

A Checkpoint of Research on Parallel I/O for High-Performance Computing

et al. 2018

View full text Add to dashboard Cite

We present a comprehensive survey on parallel I/O in the high performance computing (HPC) context. This is an important field for HPC because of the historic gap between processing power and storage latencies, which causes applications performance to be impaired when accessing or generating large amounts of data. As the available processing power and amount of data increase, I/O remains a central issue for the scientific community. In this survey, we focus on a traditional I/O stack, with a POSIX parallel file system. We present background concepts everyone could benefit from. Moreover, through the comprehensive study of publications from the most important conferences and journals in a five-year time window, we discuss the state of the art of I/O optimization approaches, access pattern extraction techniques, and performance modeling, in addition to general aspects of parallel I/O research. Through this approach, we aim at identifying the general characteristics of the field and the main current and future research topics.

show abstract

“…In this schema, the number of I/O nodes is typically larger than the number of file system servers, and smaller than the number of processing nodes. Furthermore, in this scenario, processing nodes may be powered with only a very simplified local I/O stack to avoid its interference on performance [3]. Besides performance improvements, the concept of I/O forwarding has the advantage of creating an additional layer between applications and file system.…”

Section: Introductionmentioning

confidence: 99%

TWINS: Server Access Coordination in the I/O Forwarding Layer

Bez

Boito

Schnorr

et al. 2017

2017 25th Euromicro International Conference on Parallel, Distributed and Network-Based Processing (PDP)

View full text Add to dashboard Cite

This paper presents a study of I/O scheduling techniques applied to the I/O forwarding layer. In high-performance computing environments, applications rely on parallel file systems (PFS) to obtain good I/O performance even when handling large amounts of data. To alleviate the concurrency caused by thousands of nodes accessing a significantly smaller number of PFS servers, intermediate I/O nodes are typically applied between processing nodes and the file system. Each intermediate node forwards requests from multiple clients to the system, a setup which gives this component the opportunity to perform optimizations like I/O scheduling. We evaluate scheduling techniques that improve spatiality and request size of the access patterns. We show they are only partially effective because the access pattern is not the main factor for read performance in the I/O forwarding layer. A new scheduling algorithm, TWINS, is presented to coordinate the access of intermediate I/O nodes to the data servers. Our proposal decreases concurrency at the data servers, a factor previously proven to negatively affect performance. The proposed algorithm is able to improve read performance from shared files by up to 28% over other scheduling algorithms and by up to 50% over not forwarding I/O.

show abstract

Hybrid hierarchy storage system in MilkyWay-2 supercomputer

Cited by 43 publications

References 11 publications

Paving the way for China exascale computing

Paving the way for China exascale computing

A Checkpoint of Research on Parallel I/O for High-Performance Computing

TWINS: Server Access Coordination in the I/O Forwarding Layer

Contact Info

Product

Resources

About