Chiminey: Reliable Computing and Data Management Platform in the Cloud

Abstract

Abstract-The enabling of scientific experiments that are embarrassingly parallel, long running and data-intensive into a cloud-based execution environment is a desirable, though complex undertaking for many researchers. The management of such virtual environments is cumbersome and not necessarily within the core skill set for scientists and engineers.We present here Chiminey, a software platform that enables researchers to (i) run applications on both traditional highperformance computing and cloud-based computing infrastructures, (ii) handle failure during execution, (iii) curate and visualise execution outputs, (iv) share such data with collaborators or the public, and (v) search for publicly available data.Demo video: http://youtu.be/Twi-d2WT94AI. INTRODUCTION Researchers have been using computing resources such as desktops and supercomputers for running their experiments. In order to use such resources, researchers are expected to know how to set up their execution environment, run their experiments and collect and optionally share the output of their experiments. When executing computational experiments on a local desktop machine, performing these tasks may not be challenging. However, if an experiment's resource requirements exceed those of a single workstation, then computing environments such as cluster, grid or cloud can be considered.Cloud computing [3] presents a unique opportunity for users: it enables researchers to acquire very large numbers of computing and storage resources quickly. Moreover, researchers with relatively modest requirements for parallelisation of existing code may be able to avoid learning highperformance computing (HPC) infrastructure concepts. Researchers still need to learn how to work within a cloud-based environment, which itself presents its own challenges. They need to to create and set up virtual machines (VMs) in the cloud, collect the results of their experiments, and then release the VM resources. Furthermore, cloud-based environments are more prone to failure than HPC environments due to network and third-party software issues [13], and these environments expect researchers to handle such failures themselves.The rate of technological change and innovation for compute environments is ever increasing. When a new technology

Towards Quality-Oriented Architecture

“…Another direction is to perform cloud-based testing for all abstraction levels of our framework, e.g. using Chiminey platform [24], which was created as part of the Bioscience Data Platform project [22].…”

Towards Automated Management and Analysis of Heterogeneous Data within Cannabinoids Domain

“…A model of a cloud-based platform and its opensource implementation was presented and refined in Yusuf et al (2015), Yusuf et al (2017), Spichkova et al (2016b), Spichkova et al (2015) and Spichkova et al (2016a). This model allows researchers to conduct experiments requiring complex computations over big data.…”

“…As the functionalities provided within this layer are mostly focused on researchers, it makes sense to apply the technologies that are already used in the corresponding research community. Thus, to support the analysis and visualisation of the large amount of collected data, we are going to apply a research-oriented cloud computing platform Chiminey that provides user-friendly interface for the computation/ analysis set up, as well as visualisation of the calculation results as 2D or 3D graphs, see Yusuf et al (2015) and Spichkova et al (2015). Monitoring and alerts has to be provided to notify users about updates in the data they are interested (through a previous subscription), for example, new treatment data or changes in experiments.…”

Efficient Testing based on Logical Architecture

“…Section 3.1). On this basis, we can perform testing on a cloud-based platform such as Chiminey [14], a bioscience data platform. One aim of our frameworks is to incorporate the ideas of sustainable software development and cloud-based testing into the upcoming ETSI standards [4].…”