Protocols for wide-area data-intensive applications: Design and performance issues

Abstract

Abstract-Providing high-speed data transfer is vital to various data-intensive applications. While there have been remarkable technology advances to provide ultra-high-speed network bandwidth, existing protocols and applications may not be able to fully utilize the bare-metal bandwidth due to their inefficient design. We identify the same problem remains in the field of Remote Direct Memory Access (RDMA) networks. RDMA offloads TCP/IP protocols to hardware devices. However, its benefits have not been fully exploited due to the lack of efficient software and application protocols, in particular in wide-area networks. In this paper, we address the design choices to develop such protocols. We describe a protocol implemented as part of a communication middleware. The protocol has its flow control, connection management, and task synchronization. It maximizes the parallelism of RDMA operations. We demonstrate its performance benefit on various local and wide-area testbeds, including the DOE ANI testbed with RoCE links and InfiniBand links.

Data Movement in the Internet of Things Domain

“…Data movement techniques aim to achieve this goal especially focusing on moving data at the right time to the right place, where time and place refers to the user that needs the data. Bulk data transfer protocols [12] are usually adopted to quickly transfer massive amounts of data between storage systems geographically distributed but they do not solve the problem, as the applications that need these data are highly dynamic and data movement could be required frequently and with different properties. Live data migration aims to cover this space, and the work done in [13] goes in this direction although it focuses on an environment managed by a single entity.…”

Layout-aware I/O Scheduling for terabits data movement

“…Bulk Data Movement: There have been many prior studies on the design and implementation of bulk data movement framework [1], [3], [11], [10] and their optimization in widearea networks. GridFTP [1], provided by Globus toolkit, extends the standard File Transfer Protocol (FTP), and provides high speed, reliable, and secure data transfer.…”

Efficient wide area data transfer protocols for 100 Gbps networks and beyond

“…Other related work has investigated RDMA-capable storage protocols over WANs [29] and explored system-level benefits of RDMA interfaces over 10G networks [9]. More recently there has been work to make a file transfer tool based on RDMA verbs [26]. This open source tool, called rftp, is now available.…”