Earth science data fusion with event building approach

Abstract

Objectives of the NASA Information And Data System (NAIADS) project are to develop a prototype of a conceptually new middleware framework to modernize and significantly improve efficiency of the Earth Science data fusion, big data processing and analytics. The key components of the NAIADS include: Service Oriented Architecture (SOA) multi-lingual framework, multi-sensor coincident data Predictor, fast into-memory data Staging, multi-sensor data-Event Builder, complete data-Event streaming (a workflow with minimized IO), on-line data processing control and analytics services. The NAIADS project is leveraging CLARA framework, developed in Jefferson Lab, and integrated with the ZeroMQ messaging library. The science services are prototyped and incorporated into the system. Merging the SCIAMACHY Level-1 observations and MODIS/Terra Level-2 (Clouds and Aerosols) data products, and ECMWF reanalysis will be used for NAIADS demonstration and performance tests in compute Cloud and Cluster environments.

Heterogeneous data-processing optimization with CLARA’s adaptive workflow orchestrator

“…The CLARA framework is used in CLAS12 as the main data-processing framework to design and operate event reconstruction, data analysis, and calibration, as well as data distribution and data visualization applications. The framework is also used outside of JLAB, namely at NASA [5] for MODIS and SCIAMACHY satellite data fusion and processing applications.…”

Component based dataflow processing framework

“…The NAIAD is collaborative project between JLAB and NASA to develop a prototype for the next generation Earth Science multi-sensor data fusion, processing, and analytics framework [7]. Maximizing information content by combining multi-sensor data and enabling advanced science algorithms is critical to reducing uncertainty in weather and climate research.…”