Data storage and sharing for the long tail of science

Abstract

Research data infrastructure such as storage must now accommodate new requirements resulting from trends in research data management that require researchers to store their data for the long term and make it available to other researchers. We propose Data Depot, a system and service that provides capabilities for shared space within a group, shared applications, flexible access patterns and ease of transfer at Purdue University. We evaluate Depot as a solution for storing and sharing multiterabytes of data produced in the long tail of science with a use case in soundscape ecology studies from the Human-Environment Modeling and Analysis Laboratory. We observe that with the capabilities enabled by Data Depot, researchers can easily deploy fine-grained data access control, manage data transfer and sharing, as well as integrate their workflows into a High Performance Computing environment.

Computational reproducibility of scientific workflows at extreme scales

“…Requirements for reproducibility are numerous and unclear and only started to be explored in details every step of a computational experiment (Carpen-Amarie et al, 2014). Recent recommendations include publishing source code, computational environments, and workflows in trusted repositories with persistent identifiers and links (Zhang et al, 2016a), as well as designing incentives to encourage reproducibility by journals and funding agencies (Stodden et al, 2016).…”

Clowder

“…Part of the challenge is the long tail of data, where different communities, small research teams and individual experiments have specific requirements on how the data is stored and catalogued [16]. Each community or individual scientist working on small projects produce a large portion of the total scientific output and do not have many resources to make the data accessible to the wider public [35]. With funding agencies requiring data management plans upfront [13,25], storing data on local hard drives is not feasible anymore.…”