Efficient in-network aggregation mechanism for data block repairing in data centers

Xia, Junxu; Guo, Deke; Xie, Junjie

doi:10.1016/j.future.2019.10.033

Cited by 5 publications

(2 citation statements)

References 21 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…All elements were used as inputs to the neural network model, along with the amount of change before each element. For example, the bandwidth of the following nodes and switches is shown in Equation (5) through the current time t, which is the time information, and the previous time t-1, which is the amount of change.…”

Section: Performance Of Deep Q-networkmentioning

confidence: 99%

“…To address this problem, continuous research has been conducted to optimize the network routing paths. Typically, we establish a server that can aggregate the data and reduce the number of transmission links required for transferring blocks while addressing network bottlenecks [5,6]. However, such studies have not focused on detecting dynamic bandwidths that occasionally change in actual network topologies.…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Deep Reinforcement Learning-Based Network Routing Technology for Data Recovery in Exa-Scale Cloud Distributed Clustering Systems

Shin

Kim

2021

Applied Sciences

View full text Add to dashboard Cite

Research has been conducted to efficiently transfer blocks and reduce network costs when decoding and recovering data from an erasure coding-based distributed file system. Technologies using software-defined network (SDN) controllers can collect and more efficiently manage network data. However, the bandwidth depends dynamically on the number of data transmitted on the network, and the data transfer time is inefficient owing to the longer latency of existing routing paths when nodes and switches fail. We propose deep Q-network erasure coding (DQN-EC) to solve routing problems by converging erasure coding with DQN to learn dynamically changing network elements. Using the SDN controller, DQN-EC collects the status, number, and block size of nodes possessing stored blocks during erasure coding. The fat-tree network topology used for experimental evaluation collects elements of typical network packets, the bandwidth of the nodes and switches, and other information. The data collected undergo deep reinforcement learning to avoid node and switch failures and provide optimized routing paths by selecting switches that efficiently conduct block transfers. DQN-EC achieves a 2.5-times-faster block transmission time and 0.4-times-higher network throughput than open shortest path first (OSPF) routing algorithms. The bottleneck bandwidth and transmission link cost can be reduced, improving the recovery time approximately twofold.

show abstract

Section: Performance Of Deep Q-networkmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%