Bloom Filters, Adaptivity, and the Dictionary Problem

Abstract

An approximate membership query data structure (AMQ)-such as a Bloom, quotient, or cuckoo filter-maintains a compact, probabilistic representation of a set S of keys from a universe U. It supports lookups and inserts. Some AMQs also support deletes. A query for x ∈ S returns PRESENT. A query for x ∈ S returns PRESENT with a tunable false-positive probability ε, and otherwise returns ABSENT.AMQs are widely used to speed up dictionaries that are stored remotely (e.g., on disk or across a network). The AMQ is stored locally (e.g., in memory). The remote dictionary is only accessed when the AMQ returns PRESENT. Thus, the primary performance metric of an AMQ is how often it returns ABSENT for negative queries.Existing AMQs offer weak guarantees on the number of false positives in a sequence of queries. The false-positive probability ε holds only for a single query. It is easy for an adversary to drive an AMQ's false-positive rate towards 1 by simply repeating false positives.This paper shows what it takes to get strong guarantees on the number of false positives. We say that an AMQ is adaptive if it guarantees a false-positive probability of ε for every query, regardless of answers to previous queries.We establish upper and lower bounds for adaptive AMQs. Our lower bound shows that it is impossible to build a small adaptive AMQ, even when the AMQ is immediately told whenever a query is a false positive. On the other hand, we show that it is possible to maintain an AMQ that uses the same amount of local space as a non-adaptive AMQ (up to lower order terms), performs all queries and updates in constant time, and guarantees that each negative query to the dictionary accesses remote storage with probability ε, independent of the results of past queries. Thus, we show that adaptivity can be achieved effectively for free.

Tiny Pointers

“…Proof. The starting point for our design is the adaptive filter of Bender et al [12]. Like a stash, their filter is a space-efficient internal-memory data structure that summarizes the state of an external-memory key-value dictionary.…”

“…bits, where m is the capacity of the filter. 12 The basic idea behind the adaptive filter of [12] is to store a fingerprint for each key x, where each fingerprint is taken to be some prefix of the hash h(x). Different keys have different-length fingerprints, and the invariant maintained by the filter is that no fingerprint is a prefix of any other fingerprint.…”

Optimal Ball Recycling

“…The buffered quotient filter stores newly inserted items in an in-memory quotient filter [6,7]. When the in-memory quotient filter fills, its entire contents are flushed to the on-disk quotient filter.…”

A Cost-effective and Lightweight Membership Assessment for Large-scale Data Stream

“…Simulations over real datasets show a reduction on the false positive rate in practice. In [9], the authors use similar techniques to build an adaptive oracle called the broom filters with strong theoretical guarantees. More precisely, they prove that the probability that an item is a false positive (even if S is chosen by an adversary learning from perfect oracle answers) is bounded by a constant < 1.…”

“…f) Memory partitioning: In this first simulation, we aim at determining the best way to partition a given amount of memory among our two bloom filters. Let N be the total amount of memory we can afford, then we set n = N • 0.1 • i (for i ∈ [1,9]) and n = N − n. In Figure 1, we present the results for a zipf-0.5 distribution, B = 100 and N = 500. The results are similar for others values of B or N .…”