Extending RDBMSs To Support Sparse Datasets Using An Interpreted Attribute Storage Format

Abstract

Sparse" data, in which relations have many attributes that are null for most tuples, presents a challenge for relational database management systems. If one uses the normal "horizontal" schema to store such data sets in any of the three leading commercial RDBMS, the result is tables that occupy vast amounts of storage, most of which is devoted to nulls. If one attempts to avoid this storage blowup by using a "vertical" schema, the storage utilization is indeed better, but query performance is orders of magnitude slower for certain classes of queries. In this paper, we argue that the proper way to handle sparse data is not to use a vertical schema, but rather to extend the RDBMS tuple storage format to allow the representation of sparse attributes as interpreted fields. The addition of interpreted storage allows for efficient and transparent querying of sparse data, uniform access to all attributes, and schema scalability. We show, through an implementation in PostgreSQL, that the interpreted storage approach dominates in query efficiency and ease-of-use over the current horizontal storage and vertical schema approaches over a wide range of queries and sparse data sets.

Entity Attribute Value Style Modeling Approach for Archetype Based Data

“…Beckmann et al [5] proposed a modification of physical layer (termed as interpreted attribute storage format) of row oriented RDBMS. It replaced the fixed length tuple by variable length tuple storing non-null attribute-value pairs and associated length.…”

“…In addition, NewSQL system [45] also favors to maintain ACID properties, which is inherently provided in RDBMS. However, NSM is expensive to evolve [3,5,[7][8][9][10], and is restricted up to a certain limit to avoid disk page overflow [46]. To attain the flexibility offered by NoSQL, we have adopted an approach similar to key-value approach (of NoSQL) i.e., EAV on RDBMS.…”

“…To deal with sparseness, many efforts are done at the physical layer [5,47,48], as well as the logical layer [1,10] of RDBMS. Various storage approaches suggested in RDBMS literature are detailed in the following subsections.…”

Indexing dataspaces with partitions

“…Specifically, a 3-ary vertical scheme is developed with columns including tuple identifier, attribute name, and attribute value. Beckmann et al [8] extend the RDBMS attributes to handle the sparse data as interpreted fields. A prototype implementation in the existing RDBMS is evaluated to illustrate the advanced performance in dealing with sparse data.…”

Semantic-Based RFID Data Management

“…The benchmarking system is a dual quad core 2.66GHz Intel Xeon PC, running Linux Gentoo OS, with 8 GB RAM memory, 6 MB cache memory, and a 2-disk 1Tbyte striped RAID array. We carried out our experiments using PostgreSQL 8.3, because it has been proved (see Beckmann et al in [44]) that it is significantly more efficient with respect to commercial database tools. Two different supply chains in the field of clothing production and distribution (namely Chain 20 and Chain 100 ) have been defined with 20 and 100 nodes, respectively.…”