Advanced Data Structures for Improved Cyber Resilience and Awareness in Untrusted Environments: LDRD Report

Bender, Michael A.; Berry, Jonathan W.; Farach-Colton, Martı́n; Jacobs, Justin; Johnson, Rob; Kroeger, Thomas M; Mayer, Tyler; McCauley, Samuel; Pandey, Prashant; Phillips, Cynthia A.; Porter, Alexandra; Singh, Shikha; Raizes, Justin; Xu, Haiping; Zage, David John

doi:10.2172/1528756

Cited by 3 publications

(3 citation statements)

References 0 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…In practice, this means that even a system with just a single disk can ingest hundreds of thousands of items per second. For example, at SuperComputing 2017, a single computer was easily able to maintain a B ε -tree [22] index of all connections on a 600 gigabit/sec network [8]. The system could also efficiently answer offline queries.…”

Section: Online Event Detection In External Memorymentioning

confidence: 99%

The Online Event-Detection Problem

Bender,

Berry,

Farach-Colton

et al. 2018

Preprint

Self Cite

View full text Add to dashboard Cite

Given a stream S = (s 1 , s 2 , . . . , s N ), a φ-heavy hitter is an item s i that occurs at least φN times in S. The problem of finding heavy-hitters has been extensively studied in the database literature. In this paper, we study a related problem. We say that there is a φ-event at time t if s t occurs exactly φN times in (s 1 , s 2 , . . . , s t ). Thus, for each φ-heavy hitter there is a single φ-event, which occurs when its count reaches the reporting threshold T = φN . We define the online event-detection problem (oedp) as: given φ and a stream S, report all φ-events as soon as they occur.Many real-world monitoring systems demand event detection where all events must be reported (no false negatives), in a timely manner, with no non-events reported (no false positives), and a low reporting threshold. As a result, the oedp requires a large amount of space (Ω(N ) words) and is not solvable in the streaming model or via standard sampling-based approaches.Since oedp requires large space, we focus on cache-efficient algorithms in the externalmemory model.We provide algorithms for the oedp that are within a log factor of optimal. Our algorithms are tunable: their parameters can be set to allow for bounded false-positives and a bounded delay in reporting. None of our relaxations allow false negatives since reporting all events is a strict requirement for our applications. Finally, we show improved results when the count of items in the input stream follows a power-law distribution.

show abstract

Section: Online Event Detection In External Memorymentioning

confidence: 99%

The Online Event-Detection Problem

Bender,

Berry,

Farach-Colton

et al. 2018

Preprint

Self Cite

View full text Add to dashboard Cite

show abstract

“…For example, prior work at SuperComputing 2017 showed that a single computer can easily maintain an on-disk B εtree [24] index of all connections on a 600 gigabit/sec network [8]. The system could efficiently answer offline queries.…”

Section: Introductionmentioning

confidence: 99%

“…Data structure configuration: RAM level: 8388608 slots in the CQF, levels: 4, growth factor: 4, level thresholds for on-disk level:(2,4,8), cones: 8, threads: 8, number of observations: 512M.…”

mentioning

confidence: 99%

Timely Reporting of Heavy Hitters using External Memory

Pandey

Singh

Bender

et al. 2020

Proceedings of the 2020 ACM SIGMOD International Conference on Management of Data

Self Cite

View full text Add to dashboard Cite

Given an input stream of size N , a ϕ-heavy hitter is an item that occurs at least ϕN times in S. The problem of finding heavy-hitters is extensively studied in the database literature. We study a real-time heavy-hitters variant in which an element must be reported shortly after we see its T = ϕNth occurrence (and hence becomes a heavy hitter). We call this the Timely Event Detection (TED) Problem. The TED problem models the needs of many real-world monitoring systems, which demand accurate (i.e., no false negatives) and timely reporting of all events from large, high-speed streams, and with a low reporting threshold (high sensitivity). Like the classic heavy-hitters problem, solving the TED problem without false-positives requires large space (Ω(N) words). Thus in-RAM heavy-hitters algorithms typically sacrifice accuracy (i.e., allow false positives), sensitivity, or timeliness (i.e., use multiple passes). We show how to adapt heavy-hitters algorithms to external memory to solve the TED problem on large high-speed streams while guaranteeing accuracy, sensitivity, and timeliness. Our data structures are limited only by I/O-bandwidth (not latency) and support a tunable trade-off between reporting delay and I/O overhead. With a small bounded reporting delay, our algorithms incur only a logarithmic I/O overhead.

show abstract

Timely Reporting of Heavy Hitters Using External Memory

SinghShikha

PandeyPrashant

BenderMichael

et al. 2021

ACM Trans. Database Syst.

View full text Add to dashboard Cite

Given an input stream S of size N , a ɸ-heavy hitter is an item that occurs at least ɸN times in S . The problem of finding heavy-hitters is extensively studied in the database literature. We study a real-time heavy-hitters variant in which an element must be reported shortly after we see its T = ɸ N-th occurrence (and hence it becomes a heavy hitter). We call this the Timely Event Detection ( TED ) Problem. The TED problem models the needs of many real-world monitoring systems, which demand accurate (i.e., no false negatives) and timely reporting of all events from large, high-speed streams with a low reporting threshold (high sensitivity). Like the classic heavy-hitters problem, solving the TED problem without false-positives requires large space (Ω (N) words). Thus in-RAM heavy-hitters algorithms typically sacrifice accuracy (i.e., allow false positives), sensitivity, or timeliness (i.e., use multiple passes). We show how to adapt heavy-hitters algorithms to external memory to solve the TED problem on large high-speed streams while guaranteeing accuracy, sensitivity, and timeliness. Our data structures are limited only by I/O-bandwidth (not latency) and support a tunable tradeoff between reporting delay and I/O overhead. With a small bounded reporting delay, our algorithms incur only a logarithmic I/O overhead. We implement and validate our data structures empirically using the Firehose streaming benchmark. Multi-threaded versions of our structures can scale to process 11M observations per second before becoming CPU bound. In comparison, a naive adaptation of the standard heavy-hitters algorithm to external memory would be limited by the storage device’s random I/O throughput, i.e., ≈100K observations per second.

show abstract

Advanced Data Structures for Improved Cyber Resilience and Awareness in Untrusted Environments: LDRD Report

Cited by 3 publications

References 0 publications

The Online Event-Detection Problem

The Online Event-Detection Problem

Timely Reporting of Heavy Hitters using External Memory

Timely Reporting of Heavy Hitters Using External Memory

Contact Info

Product

Resources

About