Dependable computing in virtual laboratories

Abstract

Many scienti c disciplines (e.g., biology, astrophysics, particle physics, earth sciences) are shifting from in vitro to in silico research a s m o r e p h ysical processes and natural phenomena are simulated in a computer (in silico) instead of being observed (in vitro). In many of these virtual laboratories, the computations involved are very complex and long lived. Currently, users are required to manually handle almost all aspects of such computations, including their dependability. Not surprisingly, this is a major bottleneck and a signi cant source of ineciencies. To address this issue, we h a ve d e v eloped BioOpera, an extensible process support management system for virtual laboratories. In this paper, we brie y discuss the architecture and functionality of BioOpera and show h o w it can be used to e ciently manage long lived complex computations.

Parallel computing patterns for Grid workflows

“…We can distinguish between a homogeneous data partitioning strategy and a heterogenenous data partitioning strategy. In an heterogenenous execution environment, splitting the data unevenly may be required to ensure that all tasks run to completion in the same amount of time [2].…”

Efficient lineage tracking for scientific workflows

“…From our experience working with scientists in biology [2] and astrophysics [32,33], it is clear that obtaining the basic lineage is not enough. Scientists are interested in answering queries such as "What algorithms were used to derive this data set?…”

BioOpera: cluster-aware computing

“…To use BioOpera, a process describing the computation was built and then executed using clusters I, K, L. As presented in [3], BioOpera made it easier to deal with changing conditions in the cluster, ranging from user-driven interruptions (e.g. other users requesting exclusive access to the cluster) to hardware problems (e.g.…”

“…The first experiments involved a process computing the self comparison (or All vs. All) of the protein sequence database SwissProt version 38 [8] (see [3] for details). Before switching to BioOpera, this process took several months to compute, mainly due to the overhead of manually managing the computation.…”

“…BioOpera is being actively developed at the Information and Communication Systems Research Group at the Department of Computer Science of ETH Zürich, and has already been successfully used to develop and manage long-lived computations on a set of clusters [3].…”