Big Data Analytics for Earth Sciences: the EarthServer approach

Abstract

Big Data Analytics is an emerging field since massive storage and computing capabilities have been made available by advanced e-infrastructures. Earth and Environmental sciences are likely to benefit from Big Data Analytics techniques supporting the processing of the large number of Earth Observation datasets currently acquired and generated through observations and simulations. However, Earth Science data and applications present specificities in terms of relevance of the geospatial information, wide heterogeneity of data models and formats, and complexity of processing. Therefore, Big Earth Data Analytics requires specifically tailored techniques and tools. The EarthServer Big Earth Data Analytics engine offers a solution for coverage-type datasets, built around a high performance array database technology, and the adoption and enhancement of standards for service interaction (OGC WCS and WCPS). The EarthServer solution, led by the collection of requirements from scientific communities and international initiatives, provides a holistic approach that ranges from query languages and scalability up to mobile access and visualization. The result is demonstrated and validated through the development of lighthouse applications in the Marine, Geology, Atmospheric, Planetary and Cryospheric science domains.

Building an Earth Observations Data Cube: lessons learned from the Swiss Data Cube (SDC) on generating Analysis Ready Data (ARD)

“…Open Data Cube for the AGDC; RasDaMan 1 for the EODC; THREDDS for the ESDC); and using different interfaces to interact with a DC (e.g. Python API for AGDC, GEE, EAWS, ESDC; OGC WCS and WCPS for Downloaded by [Université de Genève] at 02:41 30 November 2017 EODC) (Baumann, Furtado, Ritsch, & Widmann, 1997;Baumann, Mazzetti, et al, 2016;Flach et al, 2016;Gorelick et al, 2017). This diversity of approaches asks also for a clear definition of an EO Data Cube.…”

“…scene-based file download) and lower the barriers caused by data size and related complexities in data preparation, handling, storage and analysis (CEOS, 2017). This paradigm shift is currently represented by EO Data Cubes (Baumann, Mazzetti, et al, 2016;Purss et al, 2015), an approach that is receiving increasing attention as a new solution to store, organize, manage, and analyze EO data in a way that was not possible before. Data Cubes (DC) are aiming to realize the full potential of EO data repositories by addressing Volume, Velocity, and Variety challenges, providing access to large spatio-temporal data in an analysis ready form (Baumann, 2017;Lewis et al, 2017).…”

Standards-Based Services for Big Spatio-Temporal Data

“…Partitioning can be streamlined to optimally support any given workload of access patterns. Based on this approach, databases exceeding 100 TB are running rasdaman on spatio-temporal datacubes [9].…”

Planetary Cartography and Mapping: Where We Are Today, and Where We Are Heading For?

“…The largest, and one of the most long-lasting efforts, is VESPA (Erard et al, 2014(Erard et al, , 2017, the EuroPlanet H2020 Research Infrastructure component that deals with accessibility and distribution of planetary science data from very diverse scientific domains, including a specific surface mapping task . E-Infrastructure projects with broad Earth Science focus, such as EarthServer-2, include a Planetary Science component, PlanetServer (Baumann et al, 2015;Rossi et al, 2016). Rather than focusing on data searches and discovery or on-demand processing, PlanetServer primarily uses the OGC WCPS standard (Baumann et al, 2010) to perform real-time data analytics (Rossi et al, 2016;Marco Figuera et al, 2015).…”