Torus Network Based Distributed Storage System for Massive Multimedia Contents

Kim, Chei-Yol; Kim, Dongoh; Kim, Hong-Yeon; Kim, Young-Kyun; Seo, Dae-Won

doi:10.9717/kmms.2016.19.8.1487

Cited by 3 publications

(3 citation statements)

References 2 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Studies on the optimization of EC setups refer to a setup of stripes appropriate to data I/O size during parallel I/O or efficient running of degraded I/O by diversifying the range of data encoding [49,50]. Studies on the minimization of network contention refer to minimization of network bottleneck by dividing a recovery operation into multiple parallel small suboperations [51].…”

Section: Related Workmentioning

confidence: 99%

Erasure-Coding-Based Storage and Recovery for Distributed Exascale Storage Systems

Kim

2021

Applied Sciences

View full text Add to dashboard Cite

Various techniques have been used in distributed file systems for data availability and stability. Typically, a method for storing data in a replication technique-based distributed file system is used, but due to the problem of space efficiency, an erasure-coding (EC) technique has been utilized more recently. The EC technique improves the space efficiency problem more than the replication technique does. However, the EC technique has various performance degradation factors, such as encoding and decoding and input and output (I/O) degradation. Thus, this study proposes a buffering and combining technique in which various I/O requests that occurred during encoding in an EC-based distributed file system are combined into one and processed. In addition, it proposes four recovery measures (disk input/output load distribution, random block layout, multi-thread-based parallel recovery, and matrix recycle technique) to distribute the disk input/output loads generated during decoding.

show abstract

Section: Related Workmentioning

confidence: 99%

Erasure-Coding-Based Storage and Recovery for Distributed Exascale Storage Systems

Kim

2021

Applied Sciences

View full text Add to dashboard Cite

show abstract

“…ExSTorus-FS [18] is a scale-out distributed file system based on FUSE [6] for exa-scale storage, which supports Torus-based network configurations, EC-based Hot I/O, and multiple-MDS support [17]. ExSTorus-FS was developed on the basis of MAHA-FS, which is used for providing cloud storage services of hundreds of Peta-bytes [19].…”

Section: Exstorus-fsmentioning

confidence: 99%

“…ExSTorus-FS supports K [2,4,8,16] +M [1, 2, 4] EC volumes [17] and Reed Solomon (RS) [24], Cauchy Read Solomon (CRS) [28], Liberation (LIB) [27], Advanced Vector Extensions (AVX):CRS, and AVX2:CRS using the Jerasure library 1.2 [26], and Intel ISA library 2.10 [11,12] by erasure codes. CRS and LIB are performance-optimized encoding methods that optimize RS.…”

Section: Exstorus-fsmentioning

confidence: 99%

Cost analysis of erasure coding for exa-scale storage

Kim

et al. 2018

J Supercomput

View full text Add to dashboard Cite

With the increasing demand for mass storage, research on exa-scale storage is actively underway. When the scale of storage grows to the exa-scale, the space efficiency becomes very important. To maintain the storage reliability and improve the space efficiency, we have begun to introduce erasure coding instead of replication. However, erasure coding has many I/O performance degradation factors such as Parity Calculation, degraded I/O, Data Distribution cost, etc., whereas the existing research mainly focuses on improving the performance of the Parity Calculation.In this study, we identified the issues and bottlenecks of using erasure coding in real storage. First, we measured the I/O performance of various erasure codes to find the suitable erasure codes for real storage. Next, we analyzed the execution time for each processing step when I/O was performed and the issues when erasure coding was used in storage. Finally, we predicted the cost of EC-based I/O processing in the exa-scale storage and identified the expected problems.

show abstract

Efficient techniques of parallel recovery for erasure-coding-based distributed file systems

Kim

et al. 2019

Computing

View full text Add to dashboard Cite

Torus Network Based Distributed Storage System for Massive Multimedia Contents

Cited by 3 publications

References 2 publications

Erasure-Coding-Based Storage and Recovery for Distributed Exascale Storage Systems

Erasure-Coding-Based Storage and Recovery for Distributed Exascale Storage Systems

Cost analysis of erasure coding for exa-scale storage

Efficient techniques of parallel recovery for erasure-coding-based distributed file systems

Contact Info

Product

Resources

About