Mayflower: Improving Distributed Filesystem Performance Through SDN/Filesystem Co-Design

Rizvi, Sajjad; Li, Xi; Wong, Bernard; Kazhamiaka, Fiodar; Cassell, Benjamin

doi:10.1109/icdcs.2016.39

Cited by 9 publications

(6 citation statements)

References 19 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…We also generate background traffic to simulate the real scenario. As our system is mainly an erasure-coded storage system, we add another client/server application in which the client download data from a server selected randomly, similar to the experimental simulators of T-Update [4] and Mayflower [29]. Then we set data distributions according to the Zipfian distribution [30], which is widely used for performance evaluation in storage system.…”

Section: A Experimental Environment and Experimental Designmentioning

confidence: 99%

“…The fixed periodic interval of the controller polling is set to 1s. In practice [21] [27] [29], 1s can achieve a balance between monitoring accuracy and controller overhead. The data block size is configured to 128MB (which is also the default in HDFS-RAID).…”

Section: A Experimental Environment and Experimental Designmentioning

confidence: 99%

“…By deploying OpenFlow-based centralized traffic engineering services in the data center WAN, Google B4 [17] increase link utilization to nearly 100%. Mayflower [29] uses the network measurement information collected by software-defined network technology for copying and network path selection to improve read performance. IncBricks [32] is an in-network caching system with basic computing primitives that implements an efficient process of data center network requests through programmable network device computing.…”

Section: Related Workmentioning

confidence: 99%

See 2 more Smart Citations

SDCUP: Software-Defined-Control Based Erasure-Coded Collaborative Data Update Mechanism

Chen

Zhong

2020

IEEE Access

View full text Add to dashboard Cite

The storage system often uses erasure coding to provide the necessary fault tolerance. The erasure-coded update involves data transmission and data calculation of multiple nodes. Frequent updates will cause massive communication overhead. This paper mainly considers two issues: (1) In the scenario of frequent small-size updates, there are repetitive behaviors upon update, which causes bandwidth consumption to increase exponentially as the number of update nodes increases; (2) with the increase of data scale, there are local link busy phenomena caused by unbalanced use of links during update, which can prone to bottleneck links. In order to improve the inefficient update due to network bottlenecks. We propose SDCUP, a software-defined-control collaborative update mechanism that reduce update time for erasure-coded data with network load balance. Specially, SDCUP uses the software-defined control method to select the update transmission path according to the actual link load and adjust the data flow transmission rate by monitoring the degree of network load balance periodically. To further reduce the cross-rack update traffic, SDCUP unloads the calculation to the switch to realize the data aggregation in the rack, and it parallelizes sub-update operations to efficiently and cooperatively update. To evaluate the performance of SDCUP, we conduct simulation experiments on Mininet with real-world traces. The simulation results show that SDCUP can achieve a better load balance in multiple scenarios. Compared with the other data update schemes, the proposed method can improve the system throughput by up to 21% and reduce the update time by up to 47%. INDEX TERMS Erasure coding, software-defined, data update, network load balance.

show abstract

Section: A Experimental Environment and Experimental Designmentioning

confidence: 99%

Section: A Experimental Environment and Experimental Designmentioning

confidence: 99%

Section: Related Workmentioning

confidence: 99%

See 1 more Smart Citation

SDCUP: Software-Defined-Control Based Erasure-Coded Collaborative Data Update Mechanism

Chen

Zhong

2020

IEEE Access

View full text Add to dashboard Cite

show abstract

“…Resource allocation problems can often be formulated as mathematical optimization programs [5,20,28,30,34,35,40,42,47]; the output of these programs is the allocation of resources (e.g., accelerators, servers, or network links) to each client (e.g., jobs, data shards, or traffic commodities). Unfortunately, solving these mathematical programs can be computationally expensive (Figure 1).…”

Section: Introductionmentioning

confidence: 99%

Solving Large-Scale Granular Resource Allocation Problems Efficiently with POP

Narayanan

Kazhamiaka

Abuzaid

et al. 2021

Proceedings of the ACM SIGOPS 28th Symposium on Operating Systems Principles

Self Cite

View full text Add to dashboard Cite

Resource allocation problems in many computer systems can be formulated as mathematical optimization problems. However, finding exact solutions to these problems using offthe-shelf solvers is often intractable for large problem sizes with tight SLAs, leading system designers to rely on cheap, heuristic algorithms. We observe, however, that many allocation problems are granular: they consist of a large number of clients and resources, each client requests a small fraction of the total number of resources, and clients can interchangeably use different resources. For these problems, we propose an alternative approach that reuses the original optimization problem formulation and leads to better allocations than domain-specific heuristics. Our technique, Partitioned Optimization Problems (POP), randomly splits the problem into smaller problems (with a subset of the clients and resources in the system) and coalesces the resulting sub-allocations into a global allocation for all clients. We provide theoretical and empirical evidence as to why random partitioning works well. In our experiments, POP achieves allocations within 1.5% of the optimal with orders-of-magnitude improvements in runtime compared to existing systems for cluster scheduling, traffic engineering, and load balancing.

show abstract

“…Resource allocation problems can often be formulated as linear or integer-linear programs [4,12,17,19,20,22,25,28]; the output of these programs is the allocation of resources (e.g., accelerators, servers, or network links) given to each entity (e.g., jobs, data shards, or traffic commodities). Using existing solvers for these mathematical programs can be computationally expensive at scale -the fastest solvers for linear programs are still superlinear (approximately 𝑂 (𝑛 2.5 ) [6,18], where 𝑛 is the number of problem variables), and integer programs are even more expensive.…”

Section: Introductionmentioning

confidence: 99%

Don't Give Up on Large Optimization Problems; POP Them!

Narayanan,

Kazhamiaka,

Abuzaid

et al. 2021

Preprint

Self Cite

View full text Add to dashboard Cite

Resource allocation problems in many computer systems can be formulated as mathematical optimization problems. However, finding exact solutions to these problems using off-the-shelf solvers in an online setting is often intractable for "hyper-scale" system sizes with tight SLAs, leading system designers to rely on cheap, heuristic algorithms. In this work, we explore an alternative approach that reuses the original optimization problem formulation. By splitting the original problem into smaller, more tractable problems for subsets of the system and then coalescing resulting sub-allocations into a global solution, we achieve empirically quasi-optimal (within 1.5%) performance for multiple domains with several ordersof-magnitude improvement in runtime. Deciding how to split a large problem into smaller sub-problems, and how to coalesce split allocations into a unified allocation, needs to be performed carefully in a domain-aware way. We show common principles for splitting problems effectively across a variety of tasks, including cluster scheduling, traffic engineering, and load balancing.

show abstract

Mayflower: Improving Distributed Filesystem Performance Through SDN/Filesystem Co-Design

Cited by 9 publications

References 19 publications

SDCUP: Software-Defined-Control Based Erasure-Coded Collaborative Data Update Mechanism

SDCUP: Software-Defined-Control Based Erasure-Coded Collaborative Data Update Mechanism

Solving Large-Scale Granular Resource Allocation Problems Efficiently with POP

Don't Give Up on Large Optimization Problems; POP Them!

Contact Info

Product

Resources

About