Two-level Hash/Table approach for metadata management in distributed file systems

Díaz, Antonio; Anguita, M.; Camacho, Hugo E.; Nieto, Erik; Ortega, Julio

doi:10.1007/s11227-012-0801-y

Cited by 7 publications

(3 citation statements)

References 9 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…However, because Cassandra is hashing-based, it is challenging to relocate data considering data usage or network latency. Another hash-based data distribution method, AbFS [13], handles data storage by dividing it into units called volumes and manages segments containing multiple files as hash tables. However, when the volume size changes, the entire remapping occurs, and the mapping relationship between the segment identifier and the volume changes.…”

Section: Literature Reviewmentioning

confidence: 99%

“…Methods for distributing data in a distributed storage system include table-based [4][5][6][7][8][9][10], hashbased [11][12][13][14][15][16][17][18], and subtree-based [19][20][21][22][23] methods. Table-based data distribution technology manages one central table in multiple servers, and the central table maintains data information and identifiers for data storage.…”

Section: Introductionmentioning

confidence: 99%

“…In this case, relatively high latency and low bandwidth must be considered for a Wide Area Network (WAN) connection between data centers. However, existing distributed storage system studies [4,5,[11][12][13][14][15][19][20][21] not only did not consider the network latency between data centers but also did not consider the locality of the data request. This can increase data exploration and transfer costs due to the high latency and low bandwidth of the WAN environment.…”

Section: Introductionmentioning

confidence: 99%

See 2 more Smart Citations

Data Utilization-Based Adaptive Data Management Method for Distributed Storage System in WAN Environment

Lee¹,

Kim²,

Jo³

et al. 2023

Computer Systems Science and Engineering

View full text Add to dashboard Cite

Recently, research on a distributed storage system that efficiently manages a large amount of data has been actively conducted following data production and demand increase. Physical expansion limits exist for traditional standalone storage systems, such as I/O and file system capacity. However, the existing distributed storage system does not consider where data is consumed and is more focused on data dissemination and optimizing the lookup cost of data location. And this leads to system performance degradation due to low locality occurring in a Wide Area Network (WAN) environment with high network latency. This problem hinders deploying distributed storage systems to multiple data centers over WAN. It lowers the scalability of distributed storage systems to accommodate data storage needs. This paper proposes a method for distributing data in a WAN environment considering network latency and data locality to solve this problem and increase overall system performance. The proposed distributed storage method monitors data utilization and locality to classify data temperature as hot, warm, and cold. With assigned data temperature, the proposed algorithm adaptively selects the appropriate data center and places data accordingly to overcome the excess latency from the WAN environment, leading to overall system performance degradation. This paper also conducts simulations to evaluate the proposed and existing distributed storage methods. The result shows that our proposed method reduced latency by 38% compared to the existing method. Therefore, the proposed method in this paper can be used in large-scale distributed storage systems over a WAN environment to improve latency and performance compared to existing methods, such as consistent hashing.

show abstract

Section: Literature Reviewmentioning

confidence: 99%