HPC, Cloud and Big-Data Convergent Architectures: The LEXIS Approach

Scionti, Alberto; Martinovič, Jan; Terzo, Olivier; Walter, Etienne; Levrier, Marc; Hachinger, Stephan; Magarielli, Donato; Goubier, Thierry; Louise, Stéphane; Parodi, Antonio; Murphy, Siobhan; D'Amico, Carmine; Ciccia, Simone; Danovaro, Emanuele; Lagasio, Martina; Donnat, Frédéric; Golasowski, Martin; Quintino, Tiago; Hawkes, James; Martinovič, Tomáš; Řı́ha, Lubomı́r; Slaninová, Kateřina; Serra, Stefano; Peveri, Roberto

doi:10.1007/978-3-030-22354-0_19

Cited by 7 publications

(5 citation statements)

References 12 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Building on technology from EUDAT and on advanced orchestration, the LEXIS platform has been enabling automatised cross-system workflows involving HPC and Cloud-Computing clusters at supercomputing centres [4]. However, it has not yet specifically addressed the challenge of including DBMS in such workflows, and thus of enabling supercomputing involving the most efficient DBMS with Big or Extreme Data.…”

Section: Discussionmentioning

confidence: 99%

“…All this will be strongly supported by the LEXIS3 platform from the project "Large-Scale EXecution for Industry and Society" (2019-2021, Horizon 2020 GA No. 825532, [4]), enabling easily-usable and highly efficient workflow orchestration on heterogeneous systems (Cloud, HPC and beyond), and thus addressing challenge (i).…”

Section: Pos(isgcandhepix2023)021mentioning

confidence: 99%

“…If HPC or massive GPU-based resources shall be used in addition, efficient cross-system workflow orchestration is needed. All these requirements are basically met by the LEXIS platform [4,12], which is an important basis for the developments envisaged within EXA4MIND.…”

Section: Terraviewos As a Cloud-native Environmental Computing Applic...mentioning

confidence: 99%

See 2 more Smart Citations

Wine in the Cloud, or: Smart Vineyards with a Distributed "Extreme Data Database" and Supercomputing

Harsh,

Hachinger,

Derquennes

et al. 2023

Proceedings of International Symposium on Grids &Amp; Clouds (ISGC) 2023 in Conjunction With HEPiX Spring 2023 Workshop — PoS(I

View full text Add to dashboard Cite

In this contribution, we sketch an application of Earth System Sciences and Cloud-/Big-Databased IT, which shall soon leverage European supercomputing facilities: smart viticulture, as put into practice by Terraview. TerraviewOS is a smart vineyard "operating system", allowing wine cultivators to optimise irrigation, harvesting dates and measures against plant diseases. The system relies on satellite and drone imagery as well as in-situ sensors where available. Clearly, processing behind the UI is heavily based on Cloud Computing, with some Edge Computing close to the user or High-Performance/GPU Computing components. The substantial need for computing power in TerraviewOS, in particular for training AI-based models to generate derived data products, makes the further development of some of its modules a prime use case for the EU-funded Extreme Data processing project "EXA4MIND" (Horizon Europe GA No. 101092944). Two of the strongest academic supercomputing centres in Europe take part in EXA4MIND. The collaboration to evolve the "Smart Moisture Mapper" subsystem of TerraviewOS in this context is briefly sketched. Connecting database systems, High-Performance-/Cloud-Computing systems and European Data Spaces with appropriate data access and transfer mechanisms, EXA4MIND shall demonstrate competitive advantage in scientific and enterprise data analysis needed in complex applications such as TerraviewOS.

show abstract

Section: Discussionmentioning

confidence: 99%

Section: Pos(isgcandhepix2023)021mentioning

confidence: 99%

See 1 more Smart Citation

Wine in the Cloud, or: Smart Vineyards with a Distributed "Extreme Data Database" and Supercomputing

Harsh,

Hachinger,

Derquennes

et al. 2023

Proceedings of International Symposium on Grids &Amp; Clouds (ISGC) 2023 in Conjunction With HEPiX Spring 2023 Workshop — PoS(I

View full text Add to dashboard Cite

show abstract

“…Further details on LEXIS and the systems discussed above are published in a series of conference contributions and book chapters (e.g. [5,[27][28][29]). Open software and documents developed in the project will be disseminated via GitHub, Zenodo (dedicated sections: [30,31]) and the official EU project web pages [32].…”

Section: Discussionmentioning

confidence: 99%

HPC-Cloud-Big Data Convergent Architectures and Research Data Management: The LEXIS Approach

Hachinger¹,

Martinovič²,

Terzo³

et al. 2021

Proceedings of International Symposium on Grids &Amp; Clouds 2021 — PoS(ISGC2021)

Self Cite

View full text Add to dashboard Cite

The LEXIS project (Large-scale EXecution for Industry & Society, H2020 GA825532) provides a platform for optimised execution of Cloud-HPC workflows, reducing computation time and increasing energy efficiency. The system will rely on advanced, distributed orchestration solutions (Atos YSTIA Suite, with Alien4Cloud and Yorc, based on TOSCA), the High-End Application Execution Middleware HEAppE, and new hardware capabilities for maximising efficiency in data processing, analysis and transfer (e.g. Burst Buffers with GPU-and FPGA-based data reprocessing). LEXIS handles computation tasks and data from three Pilots, based on representative and demanding HPC/Cloud-Computing use cases in Industry (SMEs) and Science: i) Simulations of complex turbomachinery and gearbox systems in Aeronautics, ii) Tsunami simulations and earthquake loss assessments which are time-constrained to enable immediate warnings and to support well-informed decisions, and iii) Weather and Climate simulations where massive amounts of in-situ data are assimilated to improve forecasts. A user-friendly LEXIS web portal, as a unique entry point, will provide access to data as well as workflow-handling and remote visualisation functionality. As part of its back-end, LEXIS builds an elaborate system for the handling of input, intermediate and result data. At its core, a Distributed Data Infrastructure (DDI) ensures the availability of LEXIS data at all participating HPC sites, which will be federated with a common LEXIS Authentication and Authorisation Infrastructure (with unified security model, user database and policies). The DDI leverages best of breed data-management solutions from EUDAT, such as B2SAFE (based on iRODS) and B2HANDLE. REST APIs on top of it will ensure a smooth interaction with LEXIS workflows and the orchestration layer. Last, but not least, the DDI will provide functionalities for Research Data Management following the FAIR principles ("Findable, Interoperable, Accessible, Reusable"), e.g. DOI acquisition, which helps to publish and disseminate open data products.

show abstract

“…The Weather and Climate pilot of the LEXIS (Large-scale EXecution for Industry & Society) H2020 project aims at developing workflows based on weather-related models to improve existing hydro-meteorological applications for severe rainfall, flash-flood, wildland forest fires, and air quality predictions [1,2]. This pilot is taking advantage of cloud solutions and large-scale geographically distributed High-Performance Computing (HPC) resources displaced in several European countries to run experiments that require the processing of a large amount of data on a dedicated platform.…”

Section: Introductionmentioning

confidence: 99%

Detecting Extreme Rainfall Events Using the WRF-ERDS Workflow: The 15 July 2020 Palermo Case Study

Mazzoglio

Parodi

2022

Water

Self Cite

View full text Add to dashboard Cite

In this work, we describe the integration of Weather and Research Forecasting (WRF) forecasts produced by CIMA Research Foundation within ITHACA Extreme Rainfall Detection System (ERDS) to increase the forecasting skills of the overall early warning system. The entire workflow is applied to the heavy rainfall event that affected the city of Palermo on 15 July 2020, causing urban flooding due to an exceptional rainfall amount of more than 130 mm recorded in about 2.5 h. This rainfall event was not properly forecasted by meteorological models operational at the time of the event, thus not allowing to issue an adequate alert over that area. The results highlight that the improvement in the quantitative precipitation scenario forecast skills, supported by the adoption of the H2020 LEXIS computing facilities and by the assimilation of in situ observations, allowed the ERDS system to improve the prediction of the peak rainfall depths, thus paving the way to the potential issuing of an alert over the Palermo area.

show abstract

HPC, Cloud and Big-Data Convergent Architectures: The LEXIS Approach

Cited by 7 publications

References 12 publications

Wine in the Cloud, or: Smart Vineyards with a Distributed "Extreme Data Database" and Supercomputing

Wine in the Cloud, or: Smart Vineyards with a Distributed "Extreme Data Database" and Supercomputing

HPC-Cloud-Big Data Convergent Architectures and Research Data Management: The LEXIS Approach

Detecting Extreme Rainfall Events Using the WRF-ERDS Workflow: The 15 July 2020 Palermo Case Study

Contact Info

Product

Resources

About