Data Storage, Access and Catalogs in gLite

Kunszt, Péter Z.; Badino, P.; Frohner, Ákos; McCance, Gavin; Nienartowicz, K.; Rocha, Ricardo; Rodrigues, Daniele Bobrowski

doi:10.1109/lgdi.2005.1612487

Cited by 7 publications

(3 citation statements)

References 1 publication

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Grid file services are used to distribute the images and to get the results: i) a file catalog service is used to locate files or replicas in every storage element in the grid, and ii) a metadata managing service is used for accounting on the images metadata as processing status and the mapping between the original image file names and their corresponding results. In the grid implementation developed for Digi-Clima, we use the LCG File Catalogue (LFC) [19] from the Worldwide Large Hadron Collider Computing Grid (WLCG), which allows maintaining a mapping between the logical file name (LFN) for each image and the physical storage URL of all of the file replicas. We also use the ARDA Metadata Grid Application (AMGA), a general purpose metadata catalogue developed by the European Middleware Initiative middleware distribution [20].…”

Section: Mpimentioning

confidence: 99%

Digi-Clima Grid: image processing and distributed computing for recovering historical climate data

Nesmachnow

Usera

Brasileiro

2015

CLEIej

View full text Add to dashboard Cite

This article describes the Digi-Clima Grid project, whose main goals are to design and implement semi-automatic techniques for digitalizing and recovering historical climate records applying parallel computing techniques over distributed computing infrastructures. The specific tool developed for image processing is described, and the implementation over grid and cloud infrastructures is reported. An experimental analysis over institutional and volunteer-based grid/cloud distributed systems demonstrate that the proposed approach is an efficient tool for recovering historical climate data. The parallel implementations allow to distribute the processing load, achieving accurate speedup values.

show abstract

Section: Mpimentioning

confidence: 99%

Digi-Clima Grid: image processing and distributed computing for recovering historical climate data

Nesmachnow

Usera

Brasileiro

2015

CLEIej

View full text Add to dashboard Cite

show abstract

“…The gLite File Transfer Service (FTS) [10] reliably performs point-to-point transfers through the SRM protocol by creating a channel between endpoints. Channels are configurable and it is possible to configure for each channel a maximum number of transfers, allowed VOs, percentage share of simultaneous transfer for each VO and priorities of transfers within the VO, so the FTS combines reliable transfers with configuration flexibility.…”

Section: Related Workmentioning

confidence: 99%

Adaptive data management in the ARC Grid middleware

Cameron

Gholami

Karpenko

et al. 2011

J. Phys.: Conf. Ser.

View full text Add to dashboard Cite

Abstract. The Advanced Resource Connector (ARC) Grid middleware was designed almost 10 years ago, and has proven to be an attractive distributed computing solution and successful in adapting to new data management and storage technologies. However, with an ever-increasing user base and scale of resources to manage, along with the introduction of more advanced data transfer protocols, some limitations in the current architecture have become apparent. The simple first-in first-out approach to data transfer leads to bottlenecks in the system, as does the built-in assumption that all data is immediately available from remote data storage. We present an entirely new data management architecture for ARC which aims to alleviate these problems, by introducing a three-layer structure. The top layer accepts incoming requests for data transfer and directs them to the middle layer, which schedules individual transfers and negotiates with various intermediate catalog and storage systems until the physical file is ready to be transferred. The lower layer performs all operations which use large amounts of bandwidth, i.e. the physical data transfer. Using such a layered structure allows more efficient use of the available bandwidth as well as enabling late-binding of jobs to data transfer slots based on a priority system. Here we describe in full detail the design and implementation of the new system.

show abstract

“…For instance, smaller files have different settings than the bigger files. To implement these channel settings DQ2 makes use of the gLite file transfer service 37, a reliable transfer service that wraps GridFTP is compatible with the SRM interface and implements the channel settings.…”

Section: Managing Large Data Sets With Dq2mentioning

confidence: 99%

Managing very large distributed data sets on a data grid

Branco

Zaluska

Roure

et al. 2010

Concurrency and Computation

View full text Add to dashboard Cite

In this work we address the management of very large data sets, which need to be stored and processed across many computing sites. The motivation for our work is the ATLAS experiment for the Large Hadron Collider (LHC), where the authors have been involved in the development of the data management middleware. This middleware, called DQ2, has been used for the last several years by the ATLAS experiment for shipping petabytes of data to research centres and universities worldwide. We describe our experience in developing and deploying DQ2 on the Worldwide LHC computing Grid, a production Grid infrastructure formed of hundreds of computing sites. From this operational experience, we have identified an important degree of uncertainty that underlies the behaviour of large Grid infrastructures. This uncertainty is subjected to a detailed analysis, leading us to present novel modelling and simulation techniques for Data Grids. In addition, we discuss what we perceive as practical limits to the development of data distribution algorithms for Data Grids given the underlying infrastructure uncertainty, and propose future research directions. MANAGING VERY LARGE DISTRIBUTED DATA SETS 1339 Figure 1. Schematic overview of the LHC accelerator.The reasons for using multiple computing sites to store and process data include cost issues and availability of resources. A single computing site requires the concentration of resources in a single location, which is not compatible with large multinational consortiums funded by various national agencies. On the contrary, the use of distributed computing resources enables data-intensive applications to make opportunistic use of remote computing resources that would otherwise not be available. This distributed computing paradigm is referred to as a Data Grid [3].Other reasons for storing and processing data across multiple sites include geo-locality and fault tolerance. Geo-locality is the placement of data closer to its users reducing the network round-trip time required for data access. Fault tolerance in this context is related to the existence of multiple copies of the data, avoiding permanent or temporary loss of access in the event of catastrophic failure at a site.In the past years, the authors have been involved in the development and operation of the distributed data management system for a data-intensive application. The distributed data management system is called DQ2 and is used by the ATLAS experiment [4], which is part of the Large Hadron Collider (LHC) project.The LHC is a high energy physics particle accelerator experiment expected to start operation during the summer of 2009 and continue in production for about 20 years. The LHC particle accelerator extends for 27 km in a ring buried 100 m underground, as illustrated in Figure 1. Along this ring, there are various detectors that observe and record the outcome of high energy proton collisions. The raw data produced by just one of the detectors, the ATLAS experiment, amounts to tens of petabytes of data per year. These data...

show abstract

Data Storage, Access and Catalogs in gLite

Cited by 7 publications

References 1 publication

Digi-Clima Grid: image processing and distributed computing for recovering historical climate data

Digi-Clima Grid: image processing and distributed computing for recovering historical climate data

Adaptive data management in the ARC Grid middleware

Managing very large distributed data sets on a data grid

Contact Info

Product

Resources

About