The Online Event-Detection Problem

Abstract

Given a stream S = (s 1 , s 2 , . . . , s N ), a φ-heavy hitter is an item s i that occurs at least φN times in S. The problem of finding heavy-hitters has been extensively studied in the database literature. In this paper, we study a related problem. We say that there is a φ-event at time t if s t occurs exactly φN times in (s 1 , s 2 , . . . , s t ). Thus, for each φ-heavy hitter there is a single φ-event, which occurs when its count reaches the reporting threshold T = φN . We define the online event-detection problem (oedp) as: given φ and a stream S, report all φ-events as soon as they occur.Many real-world monitoring systems demand event detection where all events must be reported (no false negatives), in a timely manner, with no non-events reported (no false positives), and a low reporting threshold. As a result, the oedp requires a large amount of space (Ω(N ) words) and is not solvable in the streaming model or via standard sampling-based approaches.Since oedp requires large space, we focus on cache-efficient algorithms in the externalmemory model.We provide algorithms for the oedp that are within a log factor of optimal. Our algorithms are tunable: their parameters can be set to allow for bounded false-positives and a bounded delay in reporting. None of our relaxations allow false negatives since reporting all events is a strict requirement for our applications. Finally, we show improved results when the count of items in the input stream follows a power-law distribution.