Scheduling and Transport for File Transfers on High-Speed Optical Circuits

Abstract

Scheduling resources on Grids is a well-known problem. The extension of Grids to LambdaGrids requires the scheduling of lambdas, i.e., end-to-end high-speed circuits. In this paper, we propose a scheduling heuristic for such lambdas in support of large-scale scientific applications that require high-throughput transfers of large files. We refer to this heuristic as "Varying-Bandwidth List Scheduling" (VBLS) because the scheduler returns a Time-Range-Capacity (TRC) allocation vector with varying bandwidth levels assigned for different time ranges within the duration of a transfer. The advantage of VBLS over a fixed-bandwidth allocation scheme is that it allows the scheduler to backfill any holes left in resource allocations.Enabling VBLS requires end host applications to specify the file size in their transfer requests. To characterize VBLS, we ran simulation experiments that show that VBLS performance approaches packet-switching performance. This result means that file transfers can take advantage of bandwidth that becomes available subsequent to the start of transfers, a current and critical drawback of typical fixed-bandwidth allocation schemes in circuit-switched networks. Next, we identify the key features needed in a transport protocol that works in conjunction with VBLS and develop the "Varying Bandwidth Transport Protocol" (VBTP). VBTP is a rate based flow control scheme that is coupled with Selective-ARQ based error control. Finally, the paper concludes with a discussion on the impact of transport problems on VBLS scheduling.

An efficient transportation architecture for big data movement

“…Some e-Science projects over the world need high-speed networks provided by the optical network test-beds, such as the user controlled light paths (UCLP) in CA*net4, Starlight network in Europe, circuit-switched high-speed end-to-end transport architecture (CHEETAH), dynamic resource allocation via GMPLS optical networks (DRAGON), OptiPuter, optical metro network initiative (OMNInet), 3Tnet in China, and some others [12][13][14][15][16].…”

“…In the new data transfer model, instead of transferring these data on perpacket bases immediately upon entering the network, the optical network stores the data until it find necessary, or enough network resource is available for that transfer, or delivery the data in step-by-step ways with the help of relay storage in each switching nodes. The scheduled data transfer is realized through the use of dynamic circuit switching paradigm [12][13][14][15][16][17][18].…”

Joint scheduling for optical grid applications

“…Following the idea in [14], we associate each link l L with a capacity availability function A͑t , l͒ that records the available bandwidth of link l at time t. This function is given by…”

Advance lightpath reservation for WDM networks with dynamic traffic

“…Turning towards networks with dynamic traffic, it is typical to approach the problem simply through heuristics rather than optimization due to a continuous change in state information. For instance, Veeraraghavan et al [9] describe a new scheduling algorithm called varying-bandwidth list scheduler (VBLS) that takes advantage of the delay insensitive characteristics of bulk file transfers. VBLS analyzes the available bandwidth on a channel at a given time and determines what portion of a file it can transmit in a single time slot.…”