Addressing the need for interactive, efficient, and reproducible data processing in ecology with the datacleanr R package

Abstract

Ecological research, just as all Earth System Sciences, is becoming increasingly data-rich. Tools for processing of “big data” are continuously developed to meet corresponding technical and logistical challenges. However, even at smaller scales, data sets may be challenging when best practices in data exploration, quality control and reproducibility are to be met. This can occur when conventional methods, such as generating and assessing diagnostic visualizations or tables, become unfeasible due to time and practicality constraints. Interactive processing can alleviate this issue, and is increasingly utilized to ensure that large data sets are diligently handled. However, recent interactive tools rarely enable data manipulation, may not generate reproducible outputs, or are typically data/domain-specific. We developed datacleanr, an interactive tool that facilitates best practices in data exploration, quality control (e.g., outlier assessment) and flexible processing for multiple tabular data types, including time series and georeferenced data. The package is open-source, and based on the R programming language. A key functionality of datacleanr is the “reproducible recipe”—a translation of all interactive actions into R code, which can be integrated into existing analyses pipelines. This enables researchers experienced with script-based workflows to utilize the strengths of interactive processing without sacrificing their usual work style or functionalities from other (R) packages. We demonstrate the package’s utility by addressing two common issues during data analyses, namely 1) identifying problematic structures and artefacts in hierarchically nested data, and 2) preventing excessive loss of data from ‘coarse,’ code-based filtering of time series. Ultimately, with datacleanr we aim to improve researchers’ workflows and increase confidence in and reproducibility of their results.

High vapour pressure deficit enhances turgor limitation of stem growth in an Asian tropical rainforest tree

“…The resulting $${\psi }_{\text{soil}}$$ dynamics were validated against pre‐dawn leaf water potential ($${\psi }_{\text{leaf}}$$) dynamics (Supporting Information: Figure ). All monitoring data was inspected for outliers by using the datacleanr R package (Hurley et al, 2022) in the R software (version 4.0.2; R Core Team, 2017).…”

Daytime stomatal regulation in mature temperate trees prioritizes stem rehydration at night

“…To represent the Ψ soil dynamics at the site, we averaged the values measured from all sensors. All monitoring data were inspected for outliers using the datacleanr package (Hurley et al ., 2022) in the R software environment (v.4.0.2; R Core Team, 2017).…”

Diel and seasonal stem growth responses to climatic variation are consistent across species in a subtropical tree community

“…Raw dendrometer data were checked and processed to extract radial growth with packages datacleanr (Hurley, 2022), treenetproc (Knüsel et al ., 2021) and dend R o A nalyst (Aryal et al ., 2020) in R software (R Core Team, 2020) and were then aggregated to hourly, daily and monthly scales (Kaewmano et al ., 2022). We applied the zero‐growth approach to identify periods of stem growth from raw measurements (Fig.…”