Biobanks store and catalog human biological material that is increasingly being digitized using next-generation sequencing (NGS). There is, however, a computational bottleneck, as existing software systems are not scalable and secure enough to store and process the incoming wave of genomic data from NGS machines. In the BiobankCloud project, we are building a Hadoop-based platform for the secure storage, sharing, and parallel processing of genomic data. We extended Hadoop to include support for multi-tenant studies, reduced storage requirements with erasure coding, and added support for extensible and consistent metadata. On top of Hadoop, we built a scalable scientific workflow engine featuring a proper workflow definition language focusing on simple integration and chaining of existing tools, adaptive scheduling on Apache Yarn, and support for iterative dataflows. Our platform also supports the secure sharing of data across different, distributed Hadoop clusters. The software is easily installed and comes with a user-friendly web interface for running, managing, and accessing data sets behind a secure 2-factor authentication. Initial tests have shown that the engine scales well to dozens of nodes. The entire system is open-source and includes pre-defined workflows for popular tasks in biomedical data analysis, such as variant identification, differential transcriptome analysis using RNA-Seq, and analysis of miRNA-Seq and ChIP-Seq data.

Supplementary data are available at Bioinformatics online.

Cuneiform is a minimal functional programming language for large-scale scientific data analysis. Implementing a strict black-box view on external operators and data, it allows the direct embedding of code in a variety of external languages like Python or R, provides data-parallel higher order operators for processing large partitioned data sets, allows conditionals and general recursion, and has a naturally parallelizable evaluation strategy suitable for multi-core servers and distributed execution environments like Hadoop, HTCondor, or distributed Erlang. Cuneiform has been applied in several data-intensive research areas including remote sensing, machine learning, and bioinformatics, all of which critically depend on the flexible assembly of pre-existing tools and libraries written in different languages into complex pipelines. This paper introduces the computation semantics for Cuneiform. It presents Cuneiform's abstract syntax, a simple type system, and the semantics of evaluation. Providing an unambiguous specification of the behavior of Cuneiform eases the implementation of interpreters which we showcase by providing a concise reference implementation in Erlang. The similarity of Cuneiform's syntax to the simply typed lambda calculus puts Cuneiform in perspective and allows a straightforward discussion of its design in the context of functional programming. Moreover, the simple type system allows the deduction of the language's safety up to black-box operators. Last, the formulation of the semantics also permits the verification of compilers to and from other workflow languages.

No abstract

In biological experiments, phenotype evaluation is a common challenge. In a wide variety of applications, the phenotypic features of organisms have to be measured and statistically assessed. This is especially important as differences between wild-type and mutant or treated and untreated organisms are often very subtle. Here, we propose a set of digital image transformations that implement preprocessing, feature extraction and statistical analysis of image data that is typically generated in a biological experiment. Moreover we present AgED-Analysis given Experimental Data, a software toolkit that facilitates the process of phenotypic feature evaluation from digital image data in an automatized fashion. Suitable statistical analysis and visualization is performed and controlled via a Graphical User Interface. Furthermore, the use of open data structures allows for the convenient reuse of the acquired feature data with miscellaneous data-mining software and scientific workflow systems. The functionality of this software tool is demonstrated and validated by repeating a phytohormone response experiment carried out on the fresh water alga Coleochaete scutata. The results showed that the timely and automatic processing of digital image data aides the researcher and rationalizes the formerly lengthy and, at times, error prone data evaluation in spreadsheet documents. Furthermore, the software toolkit AgED establishes a comparable evaluation standard and provides ready-to-publish graphic export facilities.