A data management and publication workflow for a large-scale, heterogeneous sensor network

Abstract

It is common for hydrology researchers to collect data using in situ sensors at high frequencies, for extended durations, and with spatial distributions that produce data volumes requiring infrastructure for data storage, management, and sharing. The availability and utility of these data in addressing scientific questions related to water availability, water quality, and natural disasters relies on effective cyberinfrastructure that facilitates transformation of raw sensor data into usable data products. It also depends on the ability of researchers to share and access the data in useable formats. In this paper, we describe a data management and publication workflow and software tools for research groups and sites conducting long-term monitoring using in situ sensors. Functionality includes the ability to track monitoring equipment inventory and events related to field maintenance. Linking this information to the observational data is imperative in ensuring the quality of sensor-based data products. We present these tools in the context of a case study for the innovative Urban Transitions and Aridregion Hydrosustainability (iUTAH) sensor network. The iUTAH monitoring network includes sensors at aquatic and terrestrial sites for continuous monitoring of common meteorological variables, snow accumulation and melt, soil moisture, surface water flow, and surface water quality. We present the overall workflow we have developed for effectively transferring data from field monitoring sites to ultimate end-users and describe the software tools we have deployed for storing, managing, and sharing the sensor data. These tools are all open source and available for others to use.

Designing and Implementing a Network for Sensing Water Quality and Hydrology across Mountain to Urban Transitions

“…For GAMUT, Jones et al . () provide a complete description of the data management cyberinfrastructure that supports the network. In short, raw data are streamed directly into operational databases and made available online in near real time.…”

“…The following subsections describe sensor maintenance, including cases that prompted us to modify our maintenance protocol when our experience revealed deficiencies in our practices. In general, technicians record field and maintenance activities on uniform field sheets as well as digitally while we develop an online equipment management system (Jones et al ., ).…”

“…The workflow for data streaming from field sensors to operational databases to dissemination via the Internet is described by Jones et al . (). We also use HydroShare, a community repository for heterogeneous resource types (http://www.hydroshare.org), to provide long‐term archival, publication, and simplified access for the following GAMUT data resources: (1) raw data in a flat CSV file for each monitoring site; (2) quality‐controlled data for each variable at each site with the script of editing steps; and (3) stage‐discharge relationships as a package consisting of individual discharge measurements, the resulting relationship, and pertinent metadata ( e.g ., iUTAH GAMUT Working Group , , b).…”

“…The alerts are programmed as stored procedures in our operational databases, which run daily and send email notifications when data screening criteria are not met. The rules currently implemented for GAMUT include checks of battery voltage range, checks for “no data” values, checks for data persistence ( e.g ., flat line), and checks to ensure data are current (Jones et al ., ) and are consistent with community recommendations for sensor data QA/QC (Campbell et al ., ; ESIP EnviroSensing Cluster, ; Integrated Ocean Observing System, ).…”

Low-Cost, Open-Source, and Low-Power: But What to Do With the Data?

“…With the development of these innovative low-cost sensing systems, however, comes the ability to generate large volumes of high-frequency data and the challenges of how to log, transmit, store, manage, and share the resulting data (Abu-Elkheir et al, 2013). Sensor data can be difficult to manage, especially as the number of sites, variables, and the time period over which observations are collected increases (Jones et al, 2015). Because Arduino microcontrollers, Raspberry Pi computers, and other systems like them are not purpose built as environmental dataloggers, one major challenge for using them in low-cost sensing applications lies in programming them to function as dataloggers (Jiang et al, 2016;Mazumdar et al, 2017).…”