Efficient end to end data exchange using configurable compression

Abstract

We explore the use of compression methods to improve the middleware-based exchange of information in interactive or collaborative distributed applications. In such applications, good compression factors must be accompanied by compression speeds suitable for the data transfer rates sustainable across network links. Our approach combines methods that continuously monitor current network and processor resources and assess compression effectiveness, with techniques that automatically choose suitable compression techniques. By integrating these techniques into middleware, there is little need for end user involvement, other than expressing the target rates of data transmission. The resulting network-and user-aware compression methods are evaluated experimentally across a range of network links and application data, the former ranging from low end links to homes, to wide-area Internet links, to high end links in intranets, the latter including both scientific (binary molecular dynamics data) and commercial (XML) data sets. Results attained demonstrate substantial improvements of this adaptive technique for data compression over non-adaptive approaches, where better compression methods are used when CPU loads are low and/or network links are slow, and where less effective and typically, faster compression techniques are used in high end network infrastructures.

High Throughput Data-Compression for Cloud Storage

“…Methods to improve the middleware-based exchange of information in interactive or collaborative distributed applications have been proposed in [21]. The proposal combines methods that continuously monitor current network and processor resources and assess compression effectiveness, deciding on the most suitable compression technique.…”

IQ‐Services: network‐aware middleware for interactive large‐data applications

“…In concurrent work, we are applying the architecture to other communication paradigms defined by the grid community, such as the M-by-N data exchanges used in remote storage, monitoring, or visualization systems [12]. In addition, we have used IQ-Services to create resource-aware communication services that apply general compression methods to data being exchanged across wide area networks [47]. Finally, in [48] and in ongoing extensions of that work, we are using IQ-Services to create adaptive remote graphical displays, for the high end 3D visual depictions required by applications like molecular dynamics.…”

Can Flight Data Recorder Memory Be Stored on the Cloud?

“…The efficiency of the sampling has been set according to the numbers in Figure 1. Obviously, this information is specific to the particular data; however, these numbers can be easily tuned if needed by sampling even a small piece of data (Wiseman, Schwan, & Widener, 2005) extracted from the original data and sending this piece of data over an unloaded line employing unloaded CPUs. It should be noted that usually the numbers being used are very close to the constants that are detailed here, so we use these constants to give an impression of the scope of numbers.…”