Gloria

Abstract

Indexing and delivering spatial data to a massive user base composed of over a billion devices around the world stretches the limits of traditional infrastructure and operational tools. For instance, offline bulk indexing and loading fall short of viable solutions when it comes to data at scale; Integration with distributed systems such as Apache Hadoop c or Spark c is sparse, while data loading is often performed in a sub-optimal fashion by relying on intermediate file formats. We present in this paper an approach toward a hybrid online/offline indexing framework called Gloria that has been running in production settings for the past year at over 350k requests per seconds with lookup latencies under 5µs. The resulting output is an in-memory key-value store and we show that by leveraging higher level MapReduce [7] constructs as defined in FlumeJava [5], Gloria can achieve large scale key-value offline indexing in a fraction of the time required by traditional datastores while maintaining similar operational performance. Gloria also provides a spatial layer based on improvements to pointer-less quadtrees [12] and locational identifiers we call shift key that reduces the nearest neighbor problem in spatial data to simple keyvalue lookups. Shift keys have shown to outperform well established solutions such as Google S2 with locational key operations.