Framework for Bringing Data Streams to the Grid

Abstract

Abstract. Data streams are a prevalent and growing source of timely data, particularly in the scientific domain. Just as it is common today to read starting conditions such as initial weather conditions, for a scientific simulation from a file, it should be equally as easy to draw starting conditions on-demand from live data streams. But efforts to date to bring streaming data to the grid have lacked generality. In this article we introduce a new model for bringing existing data streams systems onto the grid. The model is predicated on the ability to identify stream systems that meet the criteria of being a "data resource". We establish the criteria in this article, and define a grid service architecture for a data streams resource that leverages standardization efforts in the Global Grid Forum. We discuss key research issues in realizing the data streams model. We are currently developing a prototype of this architecture using our dQUOB system.

Streamflow Programming Model for Data Streaming in Scientific Workflows

“…The Stream Engine is built on the reasoning argued in [21] that data streams have increased value when centralized decision logic can exist, and not just be limited to localized decision logic. It is a global repository for the data streams system wide and all the data streams would be channeled through the Stream Engine which would make it a single stream repository against which all the stream queries could be run against.…”

Dynamic Filtering and Mining Triggers in Mesoscale Meteorology Forecasting

“…It follows the service-oriented architecture (SOA) principles, so can be a middleware tool in a larger SOA suite. Calder views a collection of streams as an unbounded data repository, a virtual stream store [11] and provides SQL query access to the resource. We believe that the value of triggers streams increases dramatically when streams are aggregated and global behavior can be interrogated.…”