A technique for moving large data sets over high-performance long distance networks

Settlemyer, Bradley W.; Dobson, Jonathan D.; Hodson, S. W.; Kuehn, Jeffery A; Poole, Stephen W; Ruwart, Thomas M.

doi:10.1109/msst.2011.5937236

Cited by 38 publications

(16 citation statements)

References 6 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…TPG is useful to explore the optimal protocol parameters and system configurations prior to the actual data movement. It is of our interest to extend TPG with more transport protocols and methods, and integrate it into existing large data transfer tools such as XDD [9] .…”

Section: Discussionmentioning

confidence: 99%

Profiling transport performance for big data transfer over dedicated channels

Yun

Rao

et al. 2015

2015 International Conference on Computing, Networking and Communications (ICNC)

Self Cite

View full text Add to dashboard Cite

The transfer of big data is increasingly supported by dedicated channels in high-performance networks. Transport protocols play a critical role in maximizing the link utilization of such high-speed connections. We propose a Transport Profile Generator (TPG) to characterize and enhance the end-to-end throughput performance of transport protocols. TPG automates the tuning of various transport-related parameters including socket options and protocol-specific configurations, and supports multiple data streams and multiple NIC-to-NIC connections. To instantiate the design of TPG, we use UDT as an example in the implementation and conduct extensive experiments of big data transfer over high-speed network channels to illustrate how existing transport protocols benefit from TPG in optimizing their performance.Index Terms-Transport profiling, big data transfer, highperformance networks.

show abstract

Section: Discussionmentioning

confidence: 99%

Profiling transport performance for big data transfer over dedicated channels

Yun

Rao

et al. 2015

2015 International Conference on Computing, Networking and Communications (ICNC)

Self Cite

View full text Add to dashboard Cite

show abstract

“…The migration of big data sets has already been researched thoroughly. The obstacles posed by migration have been approached in numerous ways, such as using high-performance networks [20], having a workload-aware strategy [21], or through a cost-minimizing approach [22]. Considering the cloud context, an additional obstacle arises as many applications have to meet strict service-level agreements (SLA), therefore the downtime of the applications needs to be limited or eliminated entirely.…”

Section: Related Workmentioning

confidence: 99%

Live datastore transformation for optimizing big data applications in cloud environments

Vanhove

Seghbroeck

Wauters

et al. 2015

2015 IFIP/IEEE International Symposium on Integrated Network Management (IM)

View full text Add to dashboard Cite

Vendor lock-in is one of the major issues preventing companies from moving their big data applications to the cloud or changing between cloud providers. A choice in provider based on used datastores can be advantageous at first, but with everchanging applications the chosen datastore may no longer be optimal after some time. Namely, applications' requirements change due to frequent updates and feature requests, and scalability issues arise as user numbers continuously evolve. In this paper we propose a framework for the live transformation of the schema and data of datastores. Using a canonical data model the framework can be easily extended for additional datastores. The framework performs the transformation on two different levels. It uses a batch layer to transform a snapshot of the datastore, while a speed layer transforms queries inserting new or updated data into the datastore. A transformation is given between MySQL and Cassandra as a proof-of-concept. We show the correctness of the transformation and provide performance results, in terms of transformation times and overhead.

show abstract

“…Bulk Data Movement: There have been many prior studies on the design and implementation of bulk data movement framework [1], [3], [11], [10] and their optimization in widearea networks. GridFTP [1], provided by Globus toolkit, extends the standard File Transfer Protocol (FTP), and provides high speed, reliable, and secure data transfer.…”

Section: Overview For Bulk Data Movementmentioning

confidence: 99%

“…These tools are useful for moving large data faster and securely from source host to remote host over the network, however little work has been done to optimize I/O performance for PFS. XDD [11] optimizes the disk I/O performance; enabling file access with direct I/Os and multiple thread for parallelism, and varying file offset ordering improve I/O access times. However, there are several key differences when compared to our work.…”

Section: Overview For Bulk Data Movementmentioning

confidence: 99%

Layout-aware I/O Scheduling for terabits data movement

Kim

Atchley

Vallée

et al. 2013

2013 IEEE International Conference on Big Data

View full text Add to dashboard Cite

Many science facilities, such as the Department of Energy's Leadership Computing Facilities and experimental facilities including the Spallation Neutron Source, Stanford Linear Accelerator Center, and Advanced Photon Source, produce massive amounts of experimental and simulation data. These data are often shared among the facilities and with collaborating institutions. Moving large datasets over the widearea network (WAN) is a major problem inhibiting collaboration. Next-generation, terabit-networks will help alleviate the problem, however, the parallel storage systems on the endsystem hosts at these institutions can become a bottleneck for terabit data movement. The parallel storage system (PFS) is shared by simulation systems, experimental systems, analysis and visualization clusters, in addition to wide-area data movers. These competing uses often induce temporary, but significant, I/O load imbalances on the storage system, which impact the performance of all the users. The problem is a serious concern because some resources are more expensive (e.g. super computers) or have time-critical deadlines (e.g. experimental data from a light source), but parallel file systems handle all requests fairly even if some storage servers are under heavy load. This paper investigates the problem of competing workloads accessing the parallel file system and how the performance of wide-area data movement can be improved in these environments. First, we study the I/O load imbalance problems using actual I/O performance data collected from the Spider storage system at the Oak Ridge Leadership Computing Facility. Second, we present I/O optimization solutions with layout-awareness on end-system hosts for bulk data movement. With our evaluation, we show that our I/O optimization techniques can avoid the I/O congested disk groups, improving storage I/O times on parallel storage systems for terabit data movement.

show abstract

A technique for moving large data sets over high-performance long distance networks

Cited by 38 publications

References 6 publications

Profiling transport performance for big data transfer over dedicated channels

Profiling transport performance for big data transfer over dedicated channels

Live datastore transformation for optimizing big data applications in cloud environments

Layout-aware I/O Scheduling for terabits data movement

Contact Info

Product

Resources

About