StreamFlow: Cross-Breeding Cloud With HPC

Colonnelli, Iacopo; Cantalupo, Barbara; Merelli, Ivan; Aldinucci, Marco

doi:10.1109/tetc.2020.3019202

Cited by 48 publications

(23 citation statements)

References 39 publications

Supporting

Mentioning

Contrasting

Unclassified

Order By: Relevance

“…The latter version will become part of a more general portal , as [ 42 ]designed for sharing and analyse Hi-C data. Thanks to the Streamflow [ 43 ] HPC4AI native Workflow Management System, the portal will make it possible to define novel analysis “Pipeline-as-a-Service” and run them in the HPC4AI or other public cloud directly addressing the reproducibility challenge in genomic research.…”

Section: Discussionmentioning

confidence: 99%

Advantages of using graph databases to explore chromatin conformation capture experiments

et al. 2021

Self Cite

View full text Add to dashboard Cite

Background High-throughput sequencing Chromosome Conformation Capture (Hi-C) allows the study of DNA interactions and 3D chromosome folding at the genome-wide scale. Usually, these data are represented as matrices describing the binary contacts among the different chromosome regions. On the other hand, a graph-based representation can be advantageous to describe the complex topology achieved by the DNA in the nucleus of eukaryotic cells. Methods Here we discuss the use of a graph database for storing and analysing data achieved by performing Hi-C experiments. The main issue is the size of the produced data and, working with a graph-based representation, the consequent necessity of adequately managing a large number of edges (contacts) connecting nodes (genes), which represents the sources of information. For this, currently available graph visualisation tools and libraries fall short with Hi-C data. The use of graph databases, instead, supports both the analysis and the visualisation of the spatial pattern present in Hi-C data, in particular for comparing different experiments or for re-mapping omics data in a space-aware context efficiently. In particular, the possibility of describing graphs through statistical indicators and, even more, the capability of correlating them through statistical distributions allows highlighting similarities and differences among different Hi-C experiments, in different cell conditions or different cell types. Results These concepts have been implemented in NeoHiC, an open-source and user-friendly web application for the progressive visualisation and analysis of Hi-C networks based on the use of the Neo4j graph database (version 3.5). Conclusion With the accumulation of more experiments, the tool will provide invaluable support to compare neighbours of genes across experiments and conditions, helping in highlighting changes in functional domains and identifying new co-organised genomic compartments.

show abstract

Section: Discussionmentioning

confidence: 99%

Advantages of using graph databases to explore chromatin conformation capture experiments

et al. 2021

Self Cite

View full text Add to dashboard Cite

show abstract

“…For this, the group is finalising an agreement with the Italian National Supercomputing Center CINECA that will actively support the group activity, which will require both training and inference of the cartesian product of networks, datasets and network parameters. A non-trivial but enabling aspect of the work will be designing and experimenting tools making it possible to bring AI workload to supercomputers and make AI experts efficiently use large scale platforms (Aldinucci et al 2018;Colonnelli et al 2002).…”

Section: Image Analysis (Ct Scans)mentioning

confidence: 99%

The CLAIRE COVID-19 initiative: approach, experiences and recommendations

et al. 2021

Self Cite

View full text Add to dashboard Cite

A volunteer effort by Artificial Intelligence (AI) researchers has shown it can deliver significant research outcomes rapidly to help tackle COVID-19. Within two months, CLAIRE's self-organising volunteers delivered the World's first comprehensive curated repository of COVID-19-related datasets useful for drug-repurposing, drafted review papers on the role CT/X-ray scan analysis and robotics could play, and progressed research in other areas. Given the pace required and nature of voluntary efforts, the teams faced a number of challenges. These offer insights in how better to prepare for future volunteer scientific efforts and large scale, data-dependent AI collaborations in general. We offer seven recommendations on how to best leverage such efforts and collaborations in the context of managing future crises.

show abstract

“…Streamflow. The StreamFlow framework [12], [13] is a container-native Workflow Management System (WMS) written in Python 3 and based on the Common Workflow Language (CWL) Standard [14]. StreamFlow has been designed around two main principles: 1) Allowing the execution of tasks in multi-container environments, in order to support concurrent execution of multiple communicating tasks in a multi-agent ecosystem; 2) Relaxing the requirement of a single shared data space, in order to allow for hybrid workflow executions on top of multi-cloud or hybrid cloud/HPC infrastructures.…”

Section: Programming Models and Toolchainsmentioning

confidence: 99%