Taking the step from meta-information to communication middleware in computational data streams

Abstract

It is our belief that network applications relying on globally distributed shared resources will increasingly adopt meta-level descriptions to describe the data streaming in the application at runtime. Our group has developed the notion of computational data streams to describe and act upon such data flows. Our work conceptualizes the data flows as database relations over which useful operations, such as querying, can be performed. This paper shows how one can make the step from a meta-level description of data flows to an actual implementation using CORBA-style event channels and binary I/O for data transport.

IQ‐Services: network‐aware middleware for interactive large‐data applications

“…Figure 2 demonstrates such functionality by placing a clientspecific XML-based, SQL-like query with the server, which then ensures that only relevant file portions and/or file attribute data are transferred. We exploit the fact that data sharing in scientific applications means transferring data on the order of GigaBytes, from one location to another, even when all the data attributes and rows might not be needed or when certain attributes can be combined to reduce the number of bytes being transferred per row [33]. Finally, as with the real-time collaborations outlined above, the amounts of data transmitted by filters can then be adjusted to match available network bandwidth.…”