Understanding query performance in Accumulo

Abstract

Open-source, BigTable-like distributed databases provide a scalable storage solution for data-intensive applications. The simple key-value storage schema provides fast record ingest and retrieval, nearly independent of the quantity of data stored. However, real applications must support non-trivial queries that require careful key design and value indexing. We study an Apache Accumulo-based big data system designed for a network situational awareness application. The application's storage schema and data retrieval requirements are analyzed. We then characterize the corresponding Accumulo performance bottlenecks. Queries are shown to be communication-bound and server-bound in different situations. Inefficiencies in the opensource communication stack and filesystem limit network and I/O performance, respectively. Additionally, in some situations, parallel clients can contend for server-side resources. Maximizing data retrieval rates for practical queries requires effective key design, indexing, and client parallelization.

Graphulo implementation of server-side sparse matrix multiply in the Accumulo database

“…The strategy in [14] also used tablet location information to determine where clients could write locally. Knowing tabletto-tablet-server assignment could likewise aid Graphulo, not only to minimize network traffic but also to partly eliminate Apache Thrift RPC serialization, which prior work has shown is a bottleneck for scans when iterator processing is light [18]. Such an enhancement would access a local tablet server by method call in place of Scanners and BatchWriters.…”

Implementing Suffix Array Algorithm Using Apache Big Table Data Implementation

“…However it is some research seems to suggests that this problem can be easily bypassed using a big data structure. The problem of the storage size and the query performances could also be explored better even if some research already been conducted [23].…”

Lustre, hadoop, accumulo

“…The peak insert rate for a single thread is typically ~100,000 entries per second. A typical single node server can reach ~500,000 entries per second using several insert threads [Sawyer 2013]. For the hypothetical Hadoop cluster described in the previous section, the peak performance would be ~100,000,000 entries per second.…”

“…IV. ACCUMULO Relational or SQL (Structured Query Language) databases[Codd 1970, Stonebraker et al 1976 have been the de facto interface to databases since the 1980s and are the bedrock of electronic transactions around the world. More recently, keyvalue stores (NoSQL databases)[Chang et al 2008] have been developed for representing large sparse tables to aid in the analysis of data for Internet search.…”