ScaleJoin: A Deterministic, Disjoint-Parallel and Skew-Resilient Stream Join

Adaptive Stream-based Shifting Bottleneck Detection in IoT-based Computing Architectures

“…Even assuming each input stream delivers its tuples in timestamp-order, it is still challenging to preserve determinism for operators that receive tuples from various streams, as they can be interleaved due to communication. A recently proposed mechanism to support deterministic processing is ScaleGate [8], [9], a parallel coordinator that efficiently supports merging of several timestamp-sorted streams, using fine-grained synchronization. It allows for an arbitrary number of reader entities to consume tuples from the merged stream while encapsulating the communication between sources and readers through defining two methods; (i) addTuple(tuple,sourceID), which allows source entity sourceID to add tuple to the Scale-Gate, and (ii) getNextReadyTuple(readerID), which provides the next tuple in the output stream that has not been consumed yet by the reader entity readerID.…”

Deterministic, elastic and real - time processing in the big data era

“…We can observe that if an entire history H is SLT -compatible, then SSG behaves deterministically, as it returns only ready tuples as ScaleGate [60]. However, describing what happens in a history that consists of both SLT -compatible and SLT -incompatible sub-histories is not straightforward.…”

“…It is very important to guarantee the deterministic processing of input tuples for applications like click stream analysis and traffic monitoring due to the fact that non-determinism can cause money loss or missed events. Deterministic stream processing is achieved by merging the timestamp-sorted tuples coming from different streams and feeding the operator with a timestamp-sorted stream of ready [60] tuples. Definition 1.…”

“…To guarantee that only ready tuples will be read by the operators, one approach is to use dedicated operators to merge incoming tuples from multiple sources such as Input Mergers [64] and SUnions [65]. ScaleGate [60] is a recently proposed data structure that adopts a different approach, without requiring the use of extra operators. ScaleGate 2 encapsulates the necessary communication between the input sources and operator's processing threads in order to decide whether a tuple is ready or not.…”

“…A stream operator's implementation is considered deterministic, if, given the same sequences of input tuples, the same sequence of output tuples will be produced, independently of the tuples' inter-arrival time. In [60] the ScaleGate data structure guarantees that data arriving from different input streaming sources are processed in the correct order by the join operator, while it has also been used for scalable streaming aggregates [61] and analytics [34]. ScaleGate stalls the processing of each tuple until it is certain that no other Queueing Threshold=1000 Queueing Threshold=5000 Queueing Threshold=10000 Queueing Threshold=20000 Fig.…”