Cloud-Native Application Validation &amp; Stress Testing through a Framework for Auto-Cluster Deployment

Astyrakakis, Nikolaos; Nikoloudakis, Yannis; Kefaloukos, Ioannis; Skianis, Charalabos; Pallis, Evangelos; Markakis, Evangelos K.

doi:10.1109/camad.2019.8858164

Cited by 2 publications

(2 citation statements)

References 19 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The number of blocks that can be decoded and recovered reaches only up to three, which is the number of parity blocks, when configured in an RS code (6,3). In other words, three out of nine blocks can be recovered during a failure, as shown in Equation (2). Blocks d* that require recovery are regenerated into new nodes by decoding the decoding matrix coefficients (β 1 , β 2 , β 3 , .…”

Section: Principle Of Erasure Codingmentioning

confidence: 99%

“…Owing to the recent development of technologies such as smartphones, IoT, artificial intelligence, and big data, large-capacity big data are being generated and utilized. We previously used a replication technology-based distributed file system to store such data; however, with the spread of cloud computing, we have recently begun storing big data more efficiently using an erasure coding-based distributed file system [1][2][3]. Replication techniques divide the original data into multiple data blocks and store them on each distributed server through n-duplex replication.…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Deep Reinforcement Learning-Based Network Routing Technology for Data Recovery in Exa-Scale Cloud Distributed Clustering Systems

Shin

Kim

2021

Applied Sciences

View full text Add to dashboard Cite

Research has been conducted to efficiently transfer blocks and reduce network costs when decoding and recovering data from an erasure coding-based distributed file system. Technologies using software-defined network (SDN) controllers can collect and more efficiently manage network data. However, the bandwidth depends dynamically on the number of data transmitted on the network, and the data transfer time is inefficient owing to the longer latency of existing routing paths when nodes and switches fail. We propose deep Q-network erasure coding (DQN-EC) to solve routing problems by converging erasure coding with DQN to learn dynamically changing network elements. Using the SDN controller, DQN-EC collects the status, number, and block size of nodes possessing stored blocks during erasure coding. The fat-tree network topology used for experimental evaluation collects elements of typical network packets, the bandwidth of the nodes and switches, and other information. The data collected undergo deep reinforcement learning to avoid node and switch failures and provide optimized routing paths by selecting switches that efficiently conduct block transfers. DQN-EC achieves a 2.5-times-faster block transmission time and 0.4-times-higher network throughput than open shortest path first (OSPF) routing algorithms. The bottleneck bandwidth and transmission link cost can be reduced, improving the recovery time approximately twofold.

show abstract

Section: Principle Of Erasure Codingmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

Deep Reinforcement Learning-Based Network Routing Technology for Data Recovery in Exa-Scale Cloud Distributed Clustering Systems

Shin

Kim

2021

Applied Sciences

View full text Add to dashboard Cite

show abstract

Performance of Communication- and Computation-Intensive SaaS on the OpenStack Cloud

2021

View full text Add to dashboard Cite

The pervasive use of cloud computing has led to many concerns, such as performance challenges in communication- and computation-intensive services on virtual cloud resources. Most evaluations of the infrastructural overhead are based on standard benchmarks. Therefore, the impact of communication issues and infrastructure services on the performance of parallel MPI-based computations remains unclear. This paper presents the performance analysis of communication- and computation-intensive software based on the discrete element method, which is deployed as a service (SaaS) on the OpenStack cloud. The performance measured on KVM-based virtual machines and Docker containers of the OpenStack cloud is compared with that obtained by using native hardware. The improved mapping of computations to multicore resources reduced the internode MPI communication by 34.4% and increased the parallel efficiency from 0.67 to 0.78, which shows the importance of communication issues. Increasing the number of parallel processes, the overhead of the cloud infrastructure increased to 13.7% and 11.2% of the software execution time on native hardware in the case of the Docker containers and KVM-based virtual machines of the OpenStack cloud, respectively. The observed overhead was mainly caused by OpenStack service processes that increased the load imbalance of parallel MPI-based SaaS.

show abstract

Cloud-Native Application Validation & Stress Testing through a Framework for Auto-Cluster Deployment

Cited by 2 publications

References 19 publications

Deep Reinforcement Learning-Based Network Routing Technology for Data Recovery in Exa-Scale Cloud Distributed Clustering Systems

Deep Reinforcement Learning-Based Network Routing Technology for Data Recovery in Exa-Scale Cloud Distributed Clustering Systems

Performance of Communication- and Computation-Intensive SaaS on the OpenStack Cloud

Contact Info

Product

Resources

About