Fast candidate generation for real-time tweet search with bloom filter chains

Abstract

The rise of social media and other forms of user-generated content have created the demand for real-time search: against a high-velocity stream of incoming documents, users desire a list of relevant results at the time the query is issued. In the context of real-time search on tweets, this work explores candidate generation in a two-stage retrieval architecture where an initial list of results is processed by a second-stage rescorer to produce the final output. We introduce Bloom filter chains, a novel extension of Bloom filters that can dynamically expand to efficiently represent an arbitrarily long and growing list of monotonically-increasing integers with a constant false positive rate. Using a collection of Bloom filter chains, a novel approximate candidate generation algorithm called BWAND is able to perform both conjunctive and disjunctive retrieval. Experiments show that our algorithm is many times faster than competitive baselines and that this increased performance does not require sacrificing end-to-end effectiveness. Our results empirically characterize the trade-off space defined by output quality, query evaluation speed, and memory footprint for this particular search architecture.

The Impact of Future Term Statistics in Real-Time Tweet Search

“…3 We found that the idf in both conditions were nearly identical across all query terms-this is not surprising considering that idf is on a log scale, and it takes substantial variations in document frequencies to have a noticeable affect on the value. However, Figure 1 shows that there is a large difference in effectiveness for a few topics: MB15, MB17, and MB35.…”

“…In this architecture, our experiments consider the candidate generation stage. Additional work has shown that end-to-end retrieval effectiveness is insensitive to the candidate generation algorithm [6,3], which means that our experiments using simple query-likelihood accurately reflect real-world conditions.…”

Characterization of the Use of Social Media in Natural Disasters: A Systematic Review

“…It requires methods and tools which can effectively extract data via APIs and then analysing these data to extract information of interest [3,10]. Social media research has introduced various methods for the effective collection of disaster related posts such as Bloom Filter Chains for real-time tweet search [9], TAKMI technology for content analysis [8] and twitter APIs -'crawl' and 'timeline' [11]. Krishnamurthy et al (2008) used these two methods, both relying on API functions provided by Twitter for the collection of large amount of data through crawl and timeline functions of Twitter.…”

Content-Aware Partial Compression for Textual Big Data Analysis in Hadoop

“…This limits the applicability of probabilistic data structures to domain-specific use only, such as Genome Sequencing. Another application of probabilistic data structures is big data queries, for instance, BWand [32] for fast query on Twitter tweets, content filtering in MapReduce programs [21], and NoSQL databases such as Google BigTable, Apache HBase and Apache Cassandra. In these cases, the probabilistic data structures are used as an indexing technique to quickly locate information in a distributed storage system.…”