Software Connector Classification and Selection for Data-Intensive Systems

Abstract

Data-intensive systems and applications transfer large volumes of data and metadata to highly distributed users separated by geographic distance and organizational boundaries. An influential element in these large volume data transfers is the selection of the appropriate software connector that satisfies user constraints on the required data distribution scenarios. Currently, this task is typically accomplished by consulting "gurus", who rely on their intuitions, at best backed by anecdotal evidence. In this paper we present a systematic approach for selecting software connectors based on eight key dimensions of data distribution that we use to represent the data distribution scenarios. Our approach, dubbed DISCO, has been implemented as a Java-based framework. The early experience with DISCO indicates good accuracy and scalability.

Scientific Software as Workflows: From Discovery to Distribution

“…Discovery in this context decribes the phase of develo niques are tested-the scientific "solution-space" is ex Software COnnectors (DISCO) decision-making framework [13], a software extension built on top of OODT that uses architectural information and data distribution metadata to autonomously decide the appropriate data movement technology to employ for a distribution scenario, or class of scenarios. In our experience, the combination of a data movement infrastructure, and a decision-making framework that is able to choose the underlying dissemination pattern and technology to satisfy scenario requirements are essential to successfully implement and design distribution workflows.…”

On the reconfiguration of software connectors

“…In a different setting, the DISCO framework for classification and selection of software connectors' [16] meets similar objectives in what concerns connectors key characteristics and users' needs expressed as data distributions.…”

Exploiting connector knowledge to efficiently disseminate highly voluminous data sets

“…In addition, DISCO allows an architect to record observations about the performance of particular connectors in a given distribution scenario. We have currently developed three connector selection algorithms (the description of which is beyond the scope of this paper -see [3] for a complete description) that leverage our connector knowledge to pair connectors to distribution scenarios: Bayesian, Scored-based, and Optimizing [4]. Below we discuss how observation and insight are captured and used in greater detail.…”

“…Though many of these technologies purport to offer similar capabilities, the danger in simply selecting one of them and integrating it into a dissemination system is that the implications of this decision are not well understood. In our recent work [4] we have shown that connectors integrated in dissemination systems required to support highly heterogeneous distribution scenarios (variations in the dimensions of a data distribution, e.g., its total volume, its number of users, the types of data that is distributed, and so on) have critical implications on its performance (efficiency, scalability, consistency, dependability). In other words, it does matter what connector is selected and clearly not at all are equally applicable in every dissemination system, let alone in every distribution scenario.…”