Remote Memory Calls

Amaro, Emmanuel; Luo, Zhihong; Ousterhout, Amy; Panda, Aurojit; Ratnasamy, Sylvia; Shenker, Scott

doi:10.1145/3422604.3425923

Cited by 13 publications

(12 citation statements)

References 27 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…An RDMA call is ultimately a DMA operation from the server's NIC to its main memory. Programmable NICs can enable new RDMA primitives [2,45], even requiring multiple DMA calls per-packet [2]. Similarly, this can open new opportunities to rethink how direct telemetry access is implemented.…”

Section: Discussionmentioning

confidence: 99%

“…An alternative approach is letting end-hosts assist in network-wide telemetry [22,46], which unfortunately requires significant investments and infrastructure changes. Switch-generated RDMA calls Recent work has shown that programmable switches can perform RDMA calls [33], and that programmable network cards are capable of expanding upon RDMA with new and customized primitives [2]. Especially FPGA network cards show great promise for highspeed custom RDMA [39,45].…”

Section: Related Workmentioning

confidence: 99%

“…Generating RDMA instructions directly from switches is possible [33], but it also raises several challenges when used for telemetry collection: (1) how to directly write in the collector memory in such a way that the data is then easily queryable? (2) how to optimize data organization inside the collector in the presence of hundreds of thousands concurrent telemetry reporters? (3) how to make the system robust to telemetry report losses while keeping limited statefulness at switches?…”

Section: Introductionmentioning

confidence: 99%

See 2 more Smart Citations

Zero-CPU Collection with Direct Telemetry Access

Langlet,

Basat,

Ramanathan

et al. 2021

Preprint

View full text Add to dashboard Cite

Programmable switches are driving a massive increase in fine-grained measurements. This puts significant pressure on telemetry collectors that have to process reports from many switches. Past research acknowledged this problem by either improving collectors' stack performance or by limiting the amount of data sent from switches. In this paper, we take a different and radical approach: switches are responsible for directly inserting queryable telemetry data into the collectors' memory, bypassing their CPU, and thereby improving their collection scalability. We propose to use a method we call direct telemetry access, where switches jointly write telemetry reports directly into the same collector's memory region, without coordination. Our solution, DART, is probabilistic, trading memory redundancy and query success probability for CPU resources at collectors. We prototype DART using commodity hardware such as P4 switches and RDMA NICs and show that we get high query success rates with a reasonable memory overhead. For example, we can collect INT path tracing information on a fat tree topology without a collector's CPU involvement while achieving 99.9% query success probability and using just 300 bytes per flow.

show abstract

Section: Discussionmentioning

confidence: 99%

Section: Related Workmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Zero-CPU Collection with Direct Telemetry Access

Langlet,

Basat,

Ramanathan

et al. 2021

Preprint

View full text Add to dashboard Cite

show abstract

“…RDMA verbs cannot support the management of complex data structures while guaranteeing no conflicts or race conditions in the presence of multiple clients accessing the same memory region. Following the spirit of past works that explored ways to extend RDMA [3], the translator can be used as an enabler for custom RDMA operations available at reporters. The translator is in charge of receiving telemetry data, aggregating it, and performing standard RDMA calls to the associated collector.…”

Section: Direct Telemetry Accessmentioning

confidence: 99%

“…In a non-congested scenario, the translator generates appropriate RDMA calls upon the reception of DTA traffic. In case of congestion, it either drops low-priority data or re-routes critical reports to its local CPU for temporary storage 3 . It further informs reporters of the congestion to limit additional non-critical telemetry data.…”

Section: Dta Translatormentioning

confidence: 99%

Direct Telemetry Access

Langlet¹,

Basat²,

Ramanathan³

et al. 2022

Preprint

View full text Add to dashboard Cite

The emergence of programmable switches allows operators to collect a vast amount of fine-grained telemetry data in real time. However, consolidating the telemetry reports at centralized collectors to gain a network-wide view poses an immense challenge. The received data has to be transported from the switches, parsed, manipulated, and inserted in queryable data structures. As the network scales, this requires excessive CPU processing. RDMA is a transport protocol that bypasses the CPU and allows extremely high data transfer rates. Yet, RDMA is not designed for telemetry collection: it requires a stateful connection, supports only a small number of concurrent writers, and has limited writing primitives, which restricts its data aggregation applicability.We introduce Direct Telemetry Access (DTA), a solution that allows fast and efficient telemetry collection, aggregation, and indexing. Our system establishes RDMA connections only from collectors' ToR switches, called translators, that process DTA reports from all other switches. DTA features novel and expressive reporting primitives such as Key-Write, Append, Sketch-Merge, and Key-Increment that allow integration of telemetry systems such as INT and others. The translators then aggregate, batch, and write the reports to collectors' memory in queryable form. As a result, our solution can collect over 100M INT reports per second, improving over Confluo, the state-of-the-art CPU-based collector, by 17x.

show abstract