Efficient, Modular Metadata Management with Loris

Abstract

With the amount of data increasing at an alarming rate, domain-specific user-level metadata management systems have emerged in several application areas to compensate for the shortcomings of file systems. Such systems provide domainspecific storage formats for performance-optimized metadata storage, search-based access interfaces for enabling declarative queries, and type-specific indexing structures for performing scalable search over metadata. In this paper, we highlight several issues that plague these user-level systems. We then show how integrating metadata management into the Loris stack solves all these problems by design. In doing so, we show how the Loris stack provides a modular framework for implementing domainspecific solutions by presenting the design of our own Loris-based metadata management system that provides 1) LSM-tree-based metadata storage, 2) an indexing infrastructure that uses LSMtrees for maintaining real-time attribute indices, and 3) scalable metadata querying using an attribute-based query language. I. INTRODUCTIONFor over four decades, file systems have treated files as a set of attributes associated with an opaque sequence of bytes, and have provided a simple hierarchical structure for organizing the files. By providing a thin veneer over devices, and by not imposing any structure on the data they store, file systems have found widespread adoption in many application areas as preferred lightweight data stores. However, this very same generality has also led to the emergence of domain-specific, user-level metadata management systems in each application area to offset several shortcomings of file systems.In the personal computing front, file systems have been used as document stores for housing a heterogeneous mix of data ranging from small text files to large multimedia files like photos, music and videos. With the amount of data stored by users increasing at an alarming rate, hierarchy-based file access and organization has lost ground to content-based access mechanisms. Most users have resorted to using attributebased or tag-based naming schemes offered by multimedia and desktop search applications for managing and searching their data. These applications essentially build a user-level metadata management system that crawls the file system periodically to extract metadata, maintains indices on the extracted metadata, and offers application-specific search interfaces to query over metadata.Modern-day enterprise storage systems house millions of files, and as each file has at least a dozen attributes (POSIX and extended attributes), these systems store an enormous amount of metadata. In addition, storage retention requirements for meeting regulatory compliance standards further

Towards a Flexible, Lightweight Virtualization Alternative

“…They impose a specific naming structure and constraints, thus adding host complexity while taking away flexibility from the domain, in terms of semantics (e.g., posix vs Win32 file deletion), configuration (e.g., access time updates or not), and the ability to optimize metadata management for application needs [52].…”

“…The basic approach is to use one object per file in the file system. Directories may be implemented as one object each, or with centralized objects [52], for example. With no hierarchy or metadata management defined at the object level, each domain may implement the file system abstractions appropriate for its user applications.…”

Examining extended and scientific metadata for scalable index designs

“…While these performed well on their test data, they focused strictly on POSIX metadata. Loris [27] and Pantheon [21] were both indexing systems tested for system metadata only. Pantheon used B-trees, which are row-based, and will face challenges with sparse data.…”

“…While there are many workload studies for file system metadata [14,9,8,20,13,28], they have focused on POSIX metadata. Search systems based on them [19,17,27,21] attempt to extrapolate performance for other use cases. By contrast, we examine scientific metadata directly, in order to better understand the design space of scientific metadata and content indexing systems.…”

MBFS: a parallel metadata search method based on Bloomfilters using MapReduce for large-scale file systems

“…However, traditional hierarchical file systems (e.g., ext4 [37] and NTFS [44]) are ill suited for search, because the metadata design has not been changed for 40 years, when file systems contained orders of magnitude of fewer files and brute-force namespace traversal was sufficient [50]. To solve this problem, metadata search has been implemented as a search application out of file systems, such as personal computer [3] and enterprise search appliances [17,29].…”