Geoweaver: Advanced Cyberinfrastructure for Managing Hybrid Geoscientific AI Workflows

Abstract

AI (artificial intelligence)-based analysis of geospatial data has gained a lot of attention. Geospatial datasets are multi-dimensional; have spatiotemporal context; exist in disparate formats; and require sophisticated AI workflows that include not only the AI algorithm training and testing, but also data preprocessing and result post-processing. This complexity poses a huge challenge when it comes to full-stack AI workflow management, as researchers often use an assortment of time-intensive manual operations to manage their projects. However, none of the existing workflow management software provides a satisfying solution on hybrid resources, full file access, data flow, code control, and provenance. This paper introduces a new system named Geoweaver to improve the efficiency of full-stack AI workflow management. It supports linking all the preprocessing, AI training and testing, and post-processing steps into a single automated workflow. To demonstrate its utility, we present a use case in which Geoweaver manages end-to-end deep learning for in-time crop mapping using Landsat data. We show how Geoweaver effectively removes the tedium of managing various scripts, code, libraries, Jupyter Notebooks, datasets, servers, and platforms, greatly reducing the time, cost, and effort researchers must spend on such AI-based workflows. The concepts demonstrated through Geoweaver serve as an important building block in the future of cyberinfrastructure for AI research.

Deep Learning Classification for Crop Types in North Dakota

“…An advanced workflow management system is used to automate the tedious large dataset processing, to enhance time efficiency and reduce scientific reproducibility issues [13], [16], [24]. We implemented the DNN crop mapping workflow in Geoweaver-an open-source web-based workflow system [25]- [27], to help us integrate distributed resources, automate the pre/postprocessing workflows and track the provenance of the final crop maps [25]. Training DNN models is a back-and-forth process and requires a lot of skills and expertise.…”

Where Is the Provenance? Ethical Replicability and Reproducibility in GIScience and Its Critical Applications

“…First, GIScience should encourage more practical demonstrations of multiuser provenance interchange that accelerate R&R (N€ ust and Pebesma 2020; Wilson et al 2020), regardless of whether the software environments are open source or commercial, workstation or cloud based, or code versus graphic block programming (Tullis et al 2015;Tullis et al 2019). Much work remains in how to compress, store, and otherwise manage provenance records throughout their life cycle (Sun et al 2020), issues that are continually emerging in discussions on GIScience workflows (OGC 2020). Even though the W3C created the powerful PROV data model (PROV-DM; Moreau and Missier 2013) almost a decade ago, GIScience applications of PROV have been very rare (Tullis et al 2019).…”

“…Even though the W3C created the powerful PROV data model (PROV-DM; Moreau and Missier 2013) almost a decade ago, GIScience applications of PROV have been very rare (Tullis et al 2019). Successful, practical demonstrations of artificial intelligence and hybrid GIScience workflow management (e.g., Sun et al 2020) challenge software engineering teams to accelerate support for provenance interchange and workflow standards and specifications from authoritative sources such as W3C and OGC.…”

GeoFairy2: A Cross-Institution Mobile Gateway to Location-Linked Data for In-Situ Decision Making

“…A large portion of the costs in reusing existing services is charged by this step. The common processes to unify the datasets include reprojection, resampling, regridding, reformatting, mosaic, merging, getting location reports, etc [48,49]. Most web services do all the preprocessing work.…”