A Metadata Catalog Service for Data Intensive Applications

Abstract

Advances in computational, storage and network technologies as well as middle ware such as the Globus Toolkit allow scientists to expand the sophistication and scope of data-intensive applications. These applications produce and analyze terabytes and petabytes of data that are distributed in millions of files or objects. To manage these large data sets efficiently, metadata or descriptive information about the data needs to be managed. There are various types of metadata, and it is likely that a range of metadata services will exist in Grid environments that are specialized for particular types of metadata cataloguing and discovery. In this paper, we present the design of a Metadata Catalog Service (MCS) that provides a mechanism for storing and accessing descriptive metadata and allows users to query for data items based on desired attributes. We describe our experience in using the MCS with several applications and present a scalability study of the service.

DEMODS: A Description Model for Data-as-a-Service

“…Descriptions about data assets in DaaSes not just include common types of metadata, such as data size, data provider, and data schemas in contemporary data catalogs [10], [11], but also consist of, e.g., APIs for accessing data assets, data contract, and pricing models. SEMF, for example, links different types of service information, including also service licensing.…”

Ontology-based information model development for science information reuse and integration

“…Knublauch [18] also examine the benefit of using ontologies to support information modeling. Finally Singh et al [19] suggest the need for ontologies to support the development of metadata catalogs for the sophisticated data-intensive applications resulting from advances in computational, storage and network technologies and data grid infrastructures.…”

Schema-Independent and Schema-Friendly Scientific Metadata Management

“…The authors of the Globus MCS catalog (which also uses name/value pairs and a RDBMS backend), noted that the storage and reconstruction of XML was found to be a bottleneck to performance [25].…”

“…SRB's MCAT [18] and the Globus MCS metadata catalog [25], are applicable across scientific domains, but neither is designed to communicate metadata based on community XML schemas. However, MCS was an early inspiration for our work.…”