INDIGO-DataCloud: a Platform to Facilitate Seamless Access to E-Infrastructures

Salomoni, Davide; Campos, Isabel; Gaido, Luciano; Lucas, Jesús Marco de; Solagna, Peter; Gomes, Jorge; Matyska, Luděk; Fuhrman, Patrick; Hardt, Marcus; Donvito, Giacinto; Dutka, Łukasz; Płóciennik, Marcin; Barbera, R.; Blanquer, Ignácio; Ceccanti, Andrea; Cetinić, Eva; David, M.; Dūma, M.; García, Álvaro López; Moltó, Germán; Orviz, Pablo; Šustr, Zdeněk; Viljoen, Matthew; Aguilar, Fernando J.; Alves, Luís M.; Antonacci, Marica; Antonelli, L. A.; Bagnasco, S.; Bonvin, Alexandre M. J. J.; Bruno, Riccardo; Chen, Y.; Costa, Alessandro; Davidović, Davor; Ertl, Benjamin; Fargetta, Marco; Fiore, Sandro; Gallozzi, S.; Kurkcuoglu, Zeynep; Iglesias, L. Lloret; Martins, J.P.; Nuzzo, Alessandra; Nassisi, Paola; Palazzo, Cosimo; Pina, J.; Sciacca, Eva; Spiga, D.; Tangaro, M. A.; Urbaniak, Michał; Vallero, S.; Wegh, Bas; Zaccolo, V.; Zambelli, Federico; Żok, Tomasz

doi:10.1007/s10723-018-9453-3

Cited by 65 publications

(50 citation statements)

References 32 publications

Supporting

Mentioning

Contrasting

Unclassified

Order By: Relevance

“…The project will enrich already existing data management services by adding missing functionalities as requested by the user communities. The project continues the effort invested by the now ended INDIGO-DataCloud project [6] in the direction of providing friendly, web-based user interfaces and mobile access to the infrastructure data management services. The project builds on the INDIGO-DataCloud achievements in the field of Quality of Services and data lifecycle management, developing smart orchestration tools to realize easily an effective policy driven data management.…”

Section: Project Objectivesmentioning

confidence: 99%

Advancements in data management services for distributed e-infrastructures: the eXtreme-DataCloud project

Cesini¹,

Donvito

Costantini³

et al. 2019

EPJ Web Conf.

View full text Add to dashboard Cite

The development of data management services capable to cope with very large data resources is a key challenge to allow the future einfrastructures to address the needs of the next generation extreme scale scientific experiments. To face this challenge, in November 2017 the H2020 eXtreme DataCloud -XDC project has been launched. Lasting for 27 months and combining the expertise of eight large European research organisations, the project aims at developing scalable technologies for federating storage resources and managing data in highly distributed computing environments. The targeted platforms are the current and next generation e-Infrastructures deployed in Europe, such as the European Open Science Cloud (EOSC), the European Grid Infrastructure (EGI), and the Worldwide LHC Computing Grid (WLCG). The project is use-case driven with a multidisciplinary approach, addressing requirements from research communities belonging to a wide range of scientific domains: High Energy Physics, Astronomy, Photon and Life Science, Medical research. XDC is aimed at implementing data management scalable services, combining already established data management and orchestration tools, to address the following high level topics: policy driven data management based on Quality-of-Service, Data Life-cycle management, smart placement of data with caching mechanisms to reduce access latency, meta-data with no predefined schema handling, execution of pre-processing applications during ingestion, data management and protection of sensitive data in distributed e-infrastructures, intelligent data placement based on access patterns. This contribution introduces the project, presents the foreseen overall architecture and the developments that are being carried on to implement the requested functionalities.EPJ Web of Conferences 214, 04044 (2019)

show abstract

Section: Project Objectivesmentioning

confidence: 99%

Advancements in data management services for distributed e-infrastructures: the eXtreme-DataCloud project

Cesini¹,

Donvito

Costantini³

et al. 2019

EPJ Web Conf.

View full text Add to dashboard Cite

show abstract

“…1, with the chosen thresholds, the three groups are clearly identified and separated. 3 Comitato di Coordinamento del Calcolo Scientifico (Scientific Computing Coordination Committee) * ∼ 32% is non INFN personnel, usually from a cohosting university department. 4 The Large Hadron Collider beauty experiment at CERN (http://lhcb-public.web.cern.ch/lhcb-public/) 5 TB-N or net TB indicates the usable capacity, expressed in TeraBytes, after the RAID and file-system overhead.…”

Section: Cost Expandability and Optimizationmentioning

confidence: 99%

“…• Evolution of Computing towards Clouds with the European projects INDIGO-DataCloud [3] for the development of the services, HNSciCloud [4] to test the use of commercial clouds for the research, EOSCpilot [5] and EOSC-hub [6] to build the European Open Science Cloud;…”

Section: Future Directionsmentioning

confidence: 99%

The INFN scientific computing infrastructure: present status and future evolution

Carlino

dell’Agnello

2019

EPJ Web Conf.

View full text Add to dashboard Cite

The INFN scientific computing infrastructure is composed of more than 30 sites, ranging from CNAF (Tier-1 for LHC and main data center for nearly 30 other experiments) and nine LHC Tier-2s, to ∼ 20 smaller sites, including LHC Tier-3s and not-LHC experiment farms. A comprehensive review of the installed resources, together with plans for the near future, has been collected during the second half of 2017, and provides a general view of the infrastructure, its costs and its potential for expansions; it also shows the general trends in software and hardware solutions utilized in a complex reality as INFN.As of the end of 2017, the total installed CPU power exceeded 800 kHS06 (∼ 80,000 cores) while the total storage net capacity was over 57 PB on disk and 97 PB on tape: the vast majority of resources (95% of cores and 95% of storage) are concentrated in the 16 largest centers. Future evolutions are explored and are towards the consolidation into big centers; this has required a rethinking of the access policies and protocols in order to enable diverse scientific communities, beyond LHC, to fruitfully exploit the INFN resources. On top of that, such an infrastructure will be used beyond INFN experiments, and will be part of the Italian infrastructure, comprising other research institutes, universities and HPC centers. * e-mail: luca.dellagnello@cnaf.infn.it 1 https://www.garr.it/en/ 2 For the role of Tier-0, Tier-1 and Tier-2 centers in the computing of the experiments at LHC see for example

show abstract

“…for all r, m in ranks do r is a rank function, m the rank multiplier 7: -If this is true, those instances should be terminated -according to some well defined rules-and the new VM should be scheduled into that freed node.…”

Section: Preemptible Instances Designmentioning

confidence: 99%

An efficient cloud scheduler design supporting preemptible instances

García

Fernández-del-Castillo

Plasencia

2019

Future Generation Computer Systems

View full text Add to dashboard Cite

Maximizing resource utilization by performing an efficient resource provisioning is a key factor for any cloud provider: commercial actors can maximize their revenues, whereas scientific and non-commercial providers can maximize their infrastructure utilization. Traditionally, batch systems have allowed data centers to fill their resources as much as possible by using backfilling and similar techniques. However, in an IaaS cloud, where virtual machines are supposed to live indefinitely, or at least as long as the user is able to pay for them, these policies are not easily implementable. In this work we present a new scheduling algorithm for IaaS providers that is able to support preemptible instances, that can be stopped by higher priority requests without introducing large modifications in the current cloud schedulers. This scheduler enables the implementation of new cloud usage and payment models that allow more efficient usage of the resources and potential new revenue sources for commercial providers. We also study the correctness and the performace overhead of the proposed scheduler agains existing solutions.

show abstract

INDIGO-DataCloud: a Platform to Facilitate Seamless Access to E-Infrastructures

Cited by 65 publications

References 32 publications

Advancements in data management services for distributed e-infrastructures: the eXtreme-DataCloud project

Advancements in data management services for distributed e-infrastructures: the eXtreme-DataCloud project

The INFN scientific computing infrastructure: present status and future evolution

An efficient cloud scheduler design supporting preemptible instances

Contact Info

Product

Resources

About