Scalable structural index construction for JSON analytics

Abstract

JavaScript Object Notation (JSON) and its variants have gained great popularity in recent years. Unfortunately, the performance of their analytics is often dragged down by the expensive JSON parsing. To address this, recent work has shown that building bitwise indices on JSON data, called structural indices , can greatly accelerate querying. Despite its promise, the existing structural index construction does not scale well as records become larger and more complex, due to its (inherently) sequential construction process and the involvement of costly memory copies that grow as the nesting level increases. To address the above issues, this work introduces Pison - a more memory-efficient structural index constructor with supports of intra-record parallelism. First, Pison features a redesign of the bottleneck step in the existing solution. The new design is not only simpler but more memory-efficient. More importantly, Pison is able to build structural indices for a single bulky record in parallel, enabled by a group of customized parallelization techniques. Finally, Pison is also optimized for better data locality, which is especially critical in the scenario of bulky record processing. Our evaluation using real-world JSON datasets shows that Pison achieves 9.8X speedup (on average) over the existing structural index construction solution for bulky records and 4.6X speedup (on average) of end-to-end performance (indexing plus querying) over a state-of-the-art SIMD-based JSON parser on a 16-core machine.

JSONSki: streaming semi-structured data with bit-parallel fast-forwarding

“…JPStream A state-of-the-art JSON streaming library [35] RapidJSON A popular conventional JSON parser from Tencent [11] simdjson A popular SIMD-based JSON parser [40] Pison A structural index-based JSON preprocessor [34] JSONSki JSON streaming with bit-parallel fast-forward…”

“…Methods. We compare JSONSki with several representative JSON tools: JPStream [35], simdjson [40], RapidJSON [11], and Pison [34]. Tables 2 and 3 list these methods and their relevant features.…”

“…First, it builds a bitmap for the given metacharacter (Line 3). The basic idea [34,40,41] is shown by function builtMetacharBitmap(char). Here, we need to remove all the pseudo-metacharacters.…”

“…With bit-parallel fast-forward, the streaming no longer needs to scan the characters one by one, instead, it processes a word-size of characters all at once. In fact, some recent works, like simdjson [40], Mison [41], and Pison [34], also leverage the bit-parallelism in JSON data processing, but their uses are limited to locating the metacharacters and they all fall into the preprocessing scheme.…”

“…Instead of creating a parse tree, Mison [41] and Pison [34] build bitmaps for colons and commas at each level of a JSON record, called leveled bitmaps. The colon bitmap helps locate the object attributes, while the comma bitmap helps find the array elements.…”