Big Data Management in Earth Observation: The German satellite data archive at the German Aerospace Center

Kiemle, Stephan; Molch, Katrin; Schropp, Stephan; Weiland, Nicolas; Mikusch, Eberhard

doi:10.1109/mgrs.2016.2541306

Cited by 14 publications

(9 citation statements)

References 0 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Big data management in EO is not just an information technology (IT) issue; the responsibilities go beyond handling large data volumes, providing high-throughput processing capacity, and making available large network bandwidths for swift data access [43]. Cloud computing has emerged as a new paradigm to provide computing as a utility service, including IaaS (infrastructure as a service), PaaS (platform as a service), and SaaS (software as a service) [44].…”

Section: Cloud Computing For Beodmentioning

confidence: 99%

Enabling the Big Earth Observation Data via Cloud Computing and DGGS: Opportunities and Challenges

Yao

Xia

et al. 2019

Remote Sensing

View full text Add to dashboard Cite

In the era of big data, the explosive growth of Earth observation data and the rapid advancement in cloud computing technology make the global-oriented spatiotemporal data simulation possible. These dual developments also provide advantageous conditions for discrete global grid systems (DGGS). DGGS are designed to portray real-world phenomena by providing a spatiotemporal unified framework on a standard discrete geospatial data structure and theoretical support to address the challenges from big data storage, processing, and analysis to visualization and data sharing. In this paper, the trinity of big Earth observation data (BEOD), cloud computing, and DGGS is proposed, and based on this trinity theory, we explore the opportunities and challenges to handle BEOD from two aspects, namely, information technology and unified data framework. Our focus is on how cloud computing and DGGS can provide an excellent solution to enable big Earth observation data. Firstly, we describe the current status and data characteristics of Earth observation data, which indicate the arrival of the era of big data in the Earth observation domain. Subsequently, we review the cloud computing technology and DGGS framework, especially the works and contributions made in the field of BEOD, including spatial cloud computing, mainstream big data platform, DGGS standards, data models, and applications. From the aforementioned views of the general introduction, the research opportunities and challenges are enumerated and discussed, including EO data management, data fusion, and grid encoding, which are concerned with analysis models and processing performance of big Earth observation data with discrete global grid systems in the cloud environment.Remote Sens. 2020, 12, 62 2 of 15 (CEOS), over 500 EO satellites have been launched in the last half-century, and more than 150 satellites will be launched in the next 12 years [2,3]. China has launched more than 60 satellites since 1970 for comprehensive observation of the Earth's systems, including HuanJing (HJ), FengYun (FY), China-brazil earth resource satellite (CBERS), and GaoFen (GF) series. From the European Space Agency (ESA), in terms of Sentinel family, a total of 50,964,670 products had been downloaded by users since the start of data access operations, with a total data volume of 41. 35 PB [4]. The big Earth observation data (BEOD) has gradually promoted the development of global industries, research institutions, and application sectors, which has had a profound impact on the study of the Earth system [5,6], contributing to human activities, environmental monitoring, and climate changes, and also provided abundant data resource for the construction of digital Earth [7][8][9].BEOD also poses challenges to uses in terms of problem complexity, automatic analysis, and processing efficiency [10][11][12][13]. Fortunately, the rapid advancement of cloud computing technology in recent years provides strong computing power, especially for the efficiency of big geospatial data management and process...

show abstract

Section: Cloud Computing For Beodmentioning

confidence: 99%

Enabling the Big Earth Observation Data via Cloud Computing and DGGS: Opportunities and Challenges

Yao

Xia

et al. 2019

Remote Sensing

View full text Add to dashboard Cite

show abstract

“…The D-SDA is operated by the Earth Observation Center (EOC) of the German Aerospace Center and is a multi-mission data management and information system covering national and international earth observation missions [13]. The data (and derived data) is stored in a large, geo-replicated cold storage data archive facility, which relies on a robotic tape library system.…”

Section: D-sdamentioning

confidence: 99%

Cold Storage Data Archives: More Than Just a Bunch of Tapes

Memishi,

Appuswamy,

Paradies

2019

Preprint

View full text Add to dashboard Cite

The abundance of available sensor and derived data from large scientific experiments, such as earth observation programs, radio astronomy sky surveys, and high-energy physics already exceeds the storage hardware globally fabricated per year. To that end, cold storage data archives are the-often overlooked-spearheads of modern big data analytics in scientific, data-intensive application domains. While high-performance data analytics has received much attention from the research community, the growing number of problems in designing and deploying cold storage archives has only received very little attention.In this paper, we take the first step towards bridging this gap in knowledge by presenting an analysis of four real-world cold storage archives from three different application domains. In doing so, we highlight (i) workload characteristics that differentiate these archives from traditional, performance-sensitive data analytics, (ii) design trade-offs involved in building cold storage systems for these archives, and (iii) deployment trade-offs with respect to migration to the public cloud. Based on our analysis, we discuss several other important research challenges that need to be addressed by the data management community.

show abstract

“…To mitigate the radiative e ect of gases, aerosols and clouds on the image data, the Level 1C data is atmospherically corrected with ATCOR4 (Richter et al, 2011) for the Level 2A product. In addition to the processing steps described above, the Level 0 product is archived in the German Satellite Data Archive (see Kiemle et al, 2014) to ensure permanent availability of the data for future reanalysis.…”

Section: Data Processingmentioning

confidence: 99%

Airborne Imaging Spectrometer HySpex

Köhler

2016

JLSRF

View full text Add to dashboard Cite

The Remote Sensing Technology Institute (IMF) of the German Aerospace Center (DLR) operates an airborne imaging spectrometer system called HySpex. Owing to its accurate calibration, the system is well suited for benchmark reference measurements and feasibility studies for Earth observation applications. The sensor also serves as simulator for the upcoming German satellite mission EnMAP. HySpex covers the spectral range from the visible and near infrared (VNIR) to the short wave infrared (SWIR) and it has been extensively characterised with numerous measurements in the IMF calibration laboratory (CHB). The HySpex instrument is made available to interested third party users through the user service Optical Airborne Remote Sensing and Calibration Homebase (OpAiRS).

show abstract

Big Data Management in Earth Observation: The German satellite data archive at the German Aerospace Center

Cited by 14 publications

References 0 publications

Enabling the Big Earth Observation Data via Cloud Computing and DGGS: Opportunities and Challenges

Enabling the Big Earth Observation Data via Cloud Computing and DGGS: Opportunities and Challenges

Cold Storage Data Archives: More Than Just a Bunch of Tapes

Airborne Imaging Spectrometer HySpex

Contact Info

Product

Resources

About