Using a data grid to automate data preparation pipelines required for regional-scale hydrologic modeling

Billah, Mirza M.; Goodall, Jonathan L.; Narayan, U.; Essawy, Bakinam T.; Lakshmi, V.; Rajasekar, Arcot; Moore, Reagan

doi:10.1016/j.envsoft.2015.12.010

Cited by 27 publications

(21 citation statements)

References 22 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Despite this availability of tools, Billah et al [14] recognized as a major challenge for data and modelling integration the lack of common file structures and semantics. This challenge has been addressed in the current work by taking inspiration from the Sensor Observation Service open standard (SOS) [15], which defines data models to represent and exchange observations derived by sensor measures.…”

Section: Data Integration Modulementioning

confidence: 99%

Open source GIS platform for water resource modelling: FREEWAT approach in the Lugano Lake

2017

View full text Add to dashboard Cite

The FREEWAT platform is an innovative Free and Open Source water resource modelling platform integrated in the QGIS geospatial software, using the SpatiaLite database, and including globally-established simulation codes from the USGS MODFLOW models family. This paper demonstrates its application to the Lugano Lake basin case study, Switzerland and Italy. Two specific modules of the platform were used to execute data integration and analyses: the Observation Analysis Tool and the Lake Package. The first one is a newly developed module facilitating the integration of time-series observations into modelling by enabling pre-and post-processing in the model environment; the latter is an existing MOD-FLOW package allowing dynamic evaluation of groundwater/lakes interaction. In the case study implementation, a participatory approach was adopted to enhance trust and acceptance of results. These show that integration of simulation codes within GIS is highly appreciated. Furthermore, its openness and freeness allow easily sharing of developed analysis and models. Stakeholders also positively evaluated the participatory process as it empowers decision making with a better understanding of model results and uncertainties. The combination of the FREE-WAT platform and the participatory approach may constitute a valuable methodology to include scientifically based analysis to be used for policy design and implementation.

show abstract

Section: Data Integration Modulementioning

confidence: 99%

Open source GIS platform for water resource modelling: FREEWAT approach in the Lugano Lake

2017

View full text Add to dashboard Cite

show abstract

“…A challenge concerns the availability of all of these data at every site selected in the modeling process; if these data are not available everywhere, then it raises the question of how best to learn about the model inputs for every required location from the available sparse data [33]. These data, when available, are normally also contained within multiple databases maintained by various data providers, with each dataset having unique data access protocols, file formats, and semantics [27,29,30]. This heterogeneity and the gaps in source data mean that to derive the input data needed to develop and run site-specific LF models, informatics and analytical tools that can combine empirical data discovery with data integration and estimation protocols will also be required [10,33,39].…”

Section: Lf Data Assemblymentioning

confidence: 99%

“…Learning parasite transmission models that take a fuller account of heterogeneous dynamics across a spatial domain is a difficult task, but the increasing availability of geolocated demographic, intervention, and disease data [23][24][25][26] together with growing advances made in computational science approaches to knowledge discovery, particularly in the areas of (1) high performance grid-based computing and programming [8,11], (2) data discovery, integration, and assembly [11,19,[27][28][29][30][31], and (3) datadriven approaches for inferring models from measurements [32][33][34][35][36][37][38], mean that simulating disease dynamics and responses to interventions effectively across heterogeneous spatially structured environments at large scales are now becoming increasingly feasible. Bayesian data-driven modeling frameworks have received considerable attention in this regard given their ability for not only facilitating the induction of a dynamical system from data, but also in the use of multiple data sources for constraining the parameters of a model to capture the local transmission features of a spatial setting [21,22,33,[39][40][41].…”

Section: Introductionmentioning

confidence: 99%

Continental-Scale, Data-Driven Predictive Assessment of Eliminating the Vector-Borne Disease, Lymphatic Filariasis, in Sub-Saharan Africa by 2020

Michael¹,

Singh²,

Mayala³

et al. 2018

Journal of Vascular Surgery: Venous and Lymphatic Disorders

View full text Add to dashboard Cite

“…Thus, the general approach demonstrated here for MODFLOW-NWT could be expanded for any workflow that can be automated and that is compatible with Docker requirements. For example, in prior work we have constructed pre-and post-processing workflows for the Variable Infiltration Capacity (VIC) hydrologic model (Liang et al, 1996b) that could directly benefit from this method for curating, packaging, and sharing resources (Billah et al, 2016c;). These containers are efficient, lightweight, self-contained packages of computational experiments that can be nearly guaranteed to be repeated or reproduced regardless of deployment issues.…”

Section: Discussionmentioning

confidence: 99%

“…Figure 3.1 shows the data processing workflow used to generate the meteorological and land surface input datasets for a VIC model simulation. This workflow consists of a sequence of 15 data processing steps, each step requiring input datasets from different sources, and many of the datasets having unique data models (Billah et al, 2016b). These scripts are written with different programming languages including Fortran 77, C, and C++.…”

Section: Variable Infiltration Capacity (Vic) Model Pre-processing Womentioning

confidence: 99%

Advancing Cyberinfrastructure for Reproducible Hydrologic Modeling

Essawy

View full text Add to dashboard Cite

Scientists have created a significant and growing collection of software tools for data manipulation, analysis, and simulation. This software includes not only computational models, but also a large collection of data pre-and post-processing tools used to support computational modeling and data analysis. This large and diverse set of scientific modeling software presents challenges to the hydrologic science community including (1) the difficulty of making scientific workflows reproducible in support of an end-to-end hydrologic modeling use case, especially when large dataset transfers are required for data processing pipelines, (2) providing adequate and accessible component-level metadata for legacy hydrologic software, and (3) ensuring scientific reproducibility when using legacy hydrologic processing software, which often has complex software dependencies. This research uses the Variable Infiltration Capacity (VIC) and MODFLOW hydrologic models as use cases in examining these challenges.This research addresses these challenges by conducting three studies. The first study explores approaches for leveraging data grid technology in hydrologic modeling to support reproducible workflows using large datasets. Its primary contribution is a general methodology for analyzing large, distributed data collections. This is accomplished by moving processing resources to the large datasets in contrast to the typical approach of moving the datasets to the processing resources. Data grid technology is used to automate data transfers and staging, in combination with automated formal publication of generated data assets.The second study advances prior efforts for formalizing model metadata in hydrology by evaluating the OntoSoft Ontology as a means for formally structuring model metadata for lower level scientific software components. The metadata evaluated describes a data pre-processing ii workflow for the VIC hydrologic model. This workflow consists of multiple software components written by different scientists over time. The analysis begins by exploring what metadata hydrologists have already captured in unstructured forms. It then shows how this metadata could be organized into structured, machine-readable metadata using the OntoSoft Ontology.Finally, the third study explores the creation of containers using Docker to more easily execute hydrologic modeling software in a computing environment. By containerizing a model with all of its dependencies, the model is self-contained and portable. This work contributes a methodology for using HydroShare and Geotrust, two new cyberinfrastructure tools under active development, to improve reproducibility in computational hydrology. HydroShare is a web-based system for sharing hydrologic data and model resources.

show abstract

Using a data grid to automate data preparation pipelines required for regional-scale hydrologic modeling

Cited by 27 publications

References 22 publications

Open source GIS platform for water resource modelling: FREEWAT approach in the Lugano Lake

Open source GIS platform for water resource modelling: FREEWAT approach in the Lugano Lake

Continental-Scale, Data-Driven Predictive Assessment of Eliminating the Vector-Borne Disease, Lymphatic Filariasis, in Sub-Saharan Africa by 2020

Advancing Cyberinfrastructure for Reproducible Hydrologic Modeling

Contact Info

Product

Resources

About