Just-In-Time Processing of Continuous Queries

Abstract

Abstract-In a data stream management system, a continuous query is processed by an execution plan consisting of multiple operators connected via the "consumer-producer" relationship, i.e., the output of an operator (the "producer") feeds to another downstream operator (the "consumer") as input. Existing techniques execute each operator separately and push all results to its consumers, without considering whether the consumers need them. Consequently, considerable CPU and memory resources are wasted on producing and storing useless intermediate results. Motivated by this, we propose just-in-time (JIT) processing, a novel methodology that enables a consumer to return feedback expressing its current demand to the producer. The latter selectively generates results based on this information. We show, through extensive experiments, that JIT achieves significant savings in terms of both CPU time and memory consumption. I. INTRODUCTIONIn typical data stream applications, including wireless sensor networks [5], road traffic monitoring [3] and publishsubscribe services [10], data continuously flow into a DSMS. Users of the DSMS pose long-running queries, whose results are incrementally evaluated as data records arrive or expire. To answer such a query, the DSMS runs an execution plan consisting of multiple basic operators (e.g., selections, joins) connected via the producer-consumer relationship, where the output of the producer comprises the input of the consumer. Besides a few top-level operators whose results are directly presented to the user, most operators generate output for the sole purpose of feeding their consumers. Figure 1a shows an example of a continuous query expressed in CQL [1]. Tuples from three streaming sources A, B and C are joined to detect a certain event. As a real-world example, an abnormal combination of readings from close-by humidity, light and temperature sensors may trigger the alarm in a factory [5]. The clause "RANGE 5 minutes" specifies that each record is alive for exactly 5 minutes, after which it expires and is purged from the system. Figure 1b illustrates

An Introduction to Duplicate Detection

“…When a new tuple arrives on S1, there are two choices: probe S2 first or probe S3 first. If S2 is expected to produce fewer matches than S3, S2 should be probed first to minimize the number of intermediate results [Golab and Özsu, 2003b;Viglas and Naughton, 2002; and "short-circuit" the join operation as soon as possible [Yang and Papadias, 2008].…”

DHTJoin: processing continuous join queries using DHT networks

“…For example, let us consider the following query plan (V W ) ((X Y ) Z) for a query of type 2 where there are nodes connected by a producer-consumer relationship, whereby a producer node generates tuples to be processed by a consumer node [40]. The query plan (see Fig.…”

Preprocessing Method for Handling Multi-Way Join Continuous Queries over Data Streams

“…특히 Aurora, Eddy [6][7][8], STREAM 등 범용 목적의 DSMS 개발과 함께 스트림 질의어, 로드 쉐 딩(load shedding), 효율적인 질의 처리 기법 등 많은 이슈 들에 대해서 연구가 진행되었다. DSMS는 무한히 들어오 는 입력 데이터들을 처리하는 것이 목적이기 때문에 기 존의 DBMS에서 사용하는 방법으로는 조인 연산이나 집 계(aggregation) 연산과 같은 블로킹 연산은 처리할 수 없 다. DSMS는 슬라이딩 윈도우(sliding window) [9]를 적용 하여 위와 같은 문제들을 처리하지만 선택조건이나 항목 선택(projection)과 같은 단일 입력 연산 [10] (그림 4(a))에 주어진 질의 에 대해서 스트림 에 새 로운 투플이 들어온 경우 (그림 5)와 같은 AVS 및 AMS 가 생성된다. 이에 대한 조인 투플 수 계산 결과는 (그림 5)에 주어진 것과 같다.…”