Congestion Control for Large-Scale RDMA Deployments

Zhu, Yibo; Eran, Haggai; Firestone, Daniel; Guo, Chuanxiong; Lipshteyn, Marina; Liron, Yehonatan; Padhye, Jitendra; Raindel, Shachar; Yahia, Mohamad Haj; Zhang, Ming

doi:10.1145/2829988.2787484

Cited by 265 publications

(148 citation statements)

References 26 publications

Supporting

Mentioning

138

Contrasting

Order By: Relevance

“…Note that of necessity our comparison is for two different host software stacks, as DCTCP runs in an optimized kernel without PFC support whereas TIMELY is used with OS-bypass messaging. We did not implement DCTCP in OS-bypass environment due to NIC firmware limitations on processing ECN feedback [48]. We set the switch ECN marking threshold to K = 80 KB.…”

Section: Small-scale Experimentsmentioning

confidence: 99%

“…TCP-Bolt [14] uses modified DCTCP algorithm within the kernel TCP stack. DCQCN [48] uses a combination of ECN markings with a QCN-inspired rate-based congestion control algorithm implemented in the NIC. Evaluations demonstrate that it addresses the HoL blocking and unfairness problems with PFC, thus making RoCE viable for largescale deployment.…”

Section: Related Workmentioning

confidence: 99%

See 1 more Smart Citation

Timely

Mittal

Lam

Dukkipati

et al. 2015

SIGCOMM Comput. Commun. Rev.

210

View full text Add to dashboard Cite

Datacenter transports aim to deliver low latency messaging together with high throughput. We show that simple packet delay, measured as round-trip times at hosts, is an effective congestion signal without the need for switch feedback. First, we show that advances in NIC hardware have made RTT measurement possible with microsecond accuracy, and that these RTTs are sufficient to estimate switch queueing. Then we describe how TIMELY can adjust transmission rates using RTT gradients to keep packet latency low while delivering high bandwidth. We implement our design in host software running over NICs with OS-bypass capabilities. We show using experiments with up to hundreds of machines on a Clos network topology that it provides excellent performance: turning on TIMELY for OS-bypass messaging over a fabric with PFC lowers 99 percentile tail latency by 9X while maintaining near line-rate throughput. Our system also outperforms DCTCP running in an optimized kernel, reducing tail latency by 13X. To the best of our knowledge, TIMELY is the first delay-based congestion control protocol for use in the datacenter, and it achieves its results despite having an order of magnitude fewer RTT signals (due to NIC offload) than earlier delay-based schemes such as Vegas.

show abstract

Section: Small-scale Experimentsmentioning

confidence: 99%

Section: Related Workmentioning

confidence: 99%

Timely

Mittal

Lam

Dukkipati

et al. 2015

SIGCOMM Comput. Commun. Rev.

210

View full text Add to dashboard Cite

show abstract

“…A distributed scheme may be implemented as part of the end-hosts, switches, or both. Some recent distributed traffic control schemes include those presented in [8], [79], [87], [99], [100].…”

Section: A Distributedmentioning

confidence: 99%

Datacenter Traffic Control: Understanding Techniques and Trade-offs

Noormohammadpour¹,

Raghavendra²

2017

Preprint

View full text Add to dashboard Cite

Datacenters provide cost-effective and flexible access to scalable compute and storage resources necessary for today's cloud computing needs. A typical datacenter is made up of thousands of servers connected with a large network and usually managed by one operator. To provide quality access to the variety of applications and services hosted on datacenters and maximize performance, it deems necessary to use datacenter networks effectively and efficiently. Datacenter traffic is often a mix of several classes with different priorities and requirements. This includes user-generated interactive traffic, traffic with deadlines, and long-running traffic. To this end, custom transport protocols and traffic management techniques have been developed to improve datacenter network performance.In this tutorial paper, we review the general architecture of datacenter networks, various topologies proposed for them, their traffic properties, general traffic control challenges in datacenters and general traffic control objectives. The purpose of this paper is to bring out the important characteristics of traffic control in datacenters and not to survey all existing solutions (as it is virtually impossible due to massive body of existing research). We hope to provide readers with a wide range of options and factors while considering a variety of traffic control mechanisms. We discuss various characteristics of datacenter traffic control including management schemes, transmission control, traffic shaping, prioritization, load balancing, multipathing, and traffic scheduling. Next, we point to several open challenges as well as new and interesting networking paradigms. At the end of this paper, we briefly review inter-datacenter networks that connect geographically dispersed datacenters which have been receiving increasing attention recently and pose interesting and novel research problems.

show abstract

“…To address this, most data center operators add some form of end-to-end congestion control to avoid triggering pause frames. Two examples are DCQCN [38] and HPCC [22]; these use sophisticated control theory to rate-limit individual flows to better manage resources in the middle of the network.…”

Section: Introductionmentioning

confidence: 99%

“…To reduce HoL-blocking, researchers and data center operators have proposed several end-to-end congestion avoidance systems that aim to reduce buffer occupancy and thus the need for PFC pause frames. Timely [26], DCQCN [38], and HPCC [22] monitor gather information from the network in response to packet transmissions to gate how quickly to send additional packets. Because earlier results have shown that DCQCN outperforms Timely [22,39], we focus our comparison on the later two.…”

Section: Introductionmentioning

confidence: 99%

Backpressure Flow Control

Goyal¹,

Shah

Sharma

et al. 2019

Proceedings of the 2019 Workshop on Buffer Sizing

View full text Add to dashboard Cite

Effective congestion control in a multi-tenant data center is becoming increasingly challenging with rapidly increasing workload demand, ever faster links, small average transfer sizes, extremely bursty traffic, limited switch buffer capacity, and one-way protocols such as RDMA. Existing deployed algorithms, such as DCQCN, are still far from optimal in many plausible scenarios, particularly for tail latency. Many operators compensate by running their networks at low average utilization, dramatically increasing costs.In this paper, we argue that we have reached the practical limits of end-to-end congestion control. Instead, we propose a new clean slate design based on hop-by-hop per-flow flow control. We show that our approach achieves near optimal tail latency behavior even under challenging conditions such as high average link utilization and in-cast cross traffic. By contrast with prior hop-by-hop schemes, our main innovation is to show that per-flow flow control can be achieved with limited metadata and packet buffering. Further, we show that our approach generalizes well to cross-data center communication.

show abstract

Congestion Control for Large-Scale RDMA Deployments

Cited by 265 publications

References 26 publications

Timely

Timely

Datacenter Traffic Control: Understanding Techniques and Trade-offs

Backpressure Flow Control

Contact Info

Product

Resources

About