IQ‐Services: network‐aware middleware for interactive large‐data applications

Cai, Zhiyu; Eisenhauer, Greg; He, Qinming; Kumar, V. Ravi; Schwan, Karsten; Wolf, Matthew

doi:10.1002/cpe.971

Cited by 6 publications

(9 citation statements)

References 31 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…This makes it possible for binary data to be inspected and modified in transit [35], and it enables the association of graph-structured data processing overlays, termed I/OGraphs [1]. With such overlays, I/O can be customized for a rich set of backend uses, including online data visualization, data storage, and transfer to remote sites, including with standard methods like GridFTP [10]. 3.…”

Section: Data Staging Servicesmentioning

confidence: 99%

DataStager: scalable data staging services for petascale applications

et al. 2010

Self Cite

View full text Add to dashboard Cite

Known challenges for petascale machines are that (1) the costs of I/O for high performance applications can be substantial, especially for output tasks like checkpointing, and (2) noise from I/O actions can inject undesirable delays into the runtimes of such codes on individual compute nodes. This paper introduces the flexible 'DataStager' framework for data staging and alternative services within that jointly address (1) and (2). Data staging services moving output data from compute nodes to staging or I/O nodes prior to storage are used to reduce I/O overheads on applications' total processing times, and explicit management of data staging offers reduced perturbation when extracting output data from a petascale machine's compute partition. Experimental evaluations of DataStager on the Cray XT machine at Oak Ridge National Laboratory establish both the necessity of intelligent data staging and the high performance of our approach, using the GTC fusion modeling code and benchmarks running on 1000+ processors.

show abstract

Section: Data Staging Servicesmentioning

confidence: 99%

DataStager: scalable data staging services for petascale applications

et al. 2010

Self Cite

View full text Add to dashboard Cite

show abstract

“…The remainder of this paper ignores other components of the IQ-Paths middleware, referring the Overlay Node reader to a more complete description of the system in [6]. We next describe the manner in which bandwidth guarantees are attained.…”

Section: Software Architecture Of the Iq-paths Middlewarementioning

confidence: 99%

“…2. It is derived from our substantial experiences with the IQ-ECho [6] high performance publish/subscribe infrastructure implementing channel-based information subscriptions. IQ-Paths leverages IQ-ECho's support for multiple transport protocols (e.g., TCP, RUDP, SCTP) and its monitoring modules for measuring desired network metrics from middleware and in cooperation with certain transport modules (e.g., RUDP).…”

Section: Software Architecture Of the Iq-paths Middlewarementioning

confidence: 99%

“…Examples include dynamically adjusting data transfer rates [6], varying compression levels in response to monitored changes in network bandwidth [34], or changing the nature of the data being sent [6,15,35]. Other research has sought to use alternative network connections or new network infrastructures to compensate for problematic connection behaviors [26].…”

Section: Introductionmentioning

confidence: 98%

See 1 more Smart Citation

IQ-Paths: Predictably High Performance Data Streams Across Dynamic Network Overlays

2007

Self Cite

View full text Add to dashboard Cite

Overlay networks are a key vehicle for delivering network and processing resources to high performance applications. For shared networks, however, to consistently deliver such resources at desired levels of performance, overlays must be managed at runtime, based on the continuous assessment and prediction of available distributed resources. Data-intensive applications, for example, must assess, predict, and judiciously use available network paths, and dynamically choose alternate or exploit concurrent paths. Otherwise, they cannot sustain the consistent levels of performance required by tasks like remote data visualization, online program steering, and remote access to high end devices. The multiplicity of data streams occurring in complex scientific workflows or in large-scale distributed collaborations exacerbate this problem, particularly when different streams have different performance requirements. This paper presents IQ-Paths, a set of techniques and their middleware realization that implement self-regulating overlay streams for data-intensive distributed applications. Self-regulation is based on (1) the dynamic and continuous assessment of the quality of each overlay path, (2) the use of online network monitoring and statistical analyses that provide probabilistic guarantees about available path bandwidth, loss rate, and RTT, and (3) selfmanagement, via an efficient packet routing and scheduling algorithm that dynamically schedules data packets to different overlay paths in accordance with their available bandwidths. IQ-Paths offers probabilistic guarantees for applicationlevel specifications of stream utility, based on statistical predictions of available network bandwidth. This affords applications with the ability, for instance, to send control or steering data across overlay paths that offer strong guarantees for future bandwidth vs. across less guaranteed paths. Experimental results presented in this paper use IQ-Paths to better handle the different kinds of data produced by two high performance applications and one multimedia application:(1) a data-driven interactive high performance code with user-defined utility requirements, (2) an adaptive overlay version of the popular Grid-FTP application, and (3) a MPEG-4 Fine-Grained Scalable layered video streaming. 130 Z. Cai, et al.

show abstract

“…These commonly used traces track the number of end users that connect to MBone sessions over time, thereby indirectly describing the network load implied by MBone use [36]. Consequently, in Figure 7, load is stated as the number of connections over time.…”

Section: B Experimentation With Applicationsmentioning

confidence: 99%

Efficient end to end data exchange using configurable compression

Wiseman

Schwan

Widener

2005

SIGOPS Oper. Syst. Rev.

Self Cite

View full text Add to dashboard Cite

We explore the use of compression methods to improve the middleware-based exchange of information in interactive or collaborative distributed applications. In such applications, good compression factors must be accompanied by compression speeds suitable for the data transfer rates sustainable across network links. Our approach combines methods that continuously monitor current network and processor resources and assess compression effectiveness, with techniques that automatically choose suitable compression techniques. By integrating these techniques into middleware, there is little need for end user involvement, other than expressing the target rates of data transmission. The resulting network-and user-aware compression methods are evaluated experimentally across a range of network links and application data, the former ranging from low end links to homes, to wide-area Internet links, to high end links in intranets, the latter including both scientific (binary molecular dynamics data) and commercial (XML) data sets. Results attained demonstrate substantial improvements of this adaptive technique for data compression over non-adaptive approaches, where better compression methods are used when CPU loads are low and/or network links are slow, and where less effective and typically, faster compression techniques are used in high end network infrastructures.

show abstract

IQ‐Services: network‐aware middleware for interactive large‐data applications

Cited by 6 publications

References 31 publications

DataStager: scalable data staging services for petascale applications

DataStager: scalable data staging services for petascale applications

IQ-Paths: Predictably High Performance Data Streams Across Dynamic Network Overlays

Efficient end to end data exchange using configurable compression

Contact Info

Product

Resources

About