Hazelcast jet

Abstract

Jet is an open-source, high-performance, distributed stream processor built at Hazelcast during the last five years. Jet was engineered with millisecond latency on the 99.99th percentile as its primary design goal. Originally Jet's purpose was to be an execution engine that performs complex business logic on top of streams generated by Hazelcast's In-memory Data Grid (IMDG): a set of in-memory, partitioned and replicated data structures. With time, Jet evolved into a full-fledged, scale-out stream processor that can handle outof-order streams and provide exactly-once processing guarantees. Jet's end-to-end latency lies in the order of milliseconds, and its throughput in the order of millions of events per CPU-core. This paper presents the main design decisions we made in order to maximize the performance per CPU-core, alongside lessons learned, and an empirical performance evaluation.

ShuffleBench: A Benchmark for Large-Scale Data Shuffling Operations with Distributed Stream Processing Frameworks

“…Popular stream processing frameworks include Apache Flink [6], Hazelcast with its Jet engine [10], Apache Kafka Streams [31,38], and Apache Spark [39] with its Structured Streaming engine [4]. Although all these frameworks follow similar concepts, several differences in their design decisions, programming functionalities, and execution models can be noted.…”

“…State-of-the-art distributed stream processing frameworks such as Spark [4,39], Flink [6], Kafka Streams [31,38], or Hazelcast with its Jet engine [10] have gained widespread adoption over the last years not only for building data analytics pipelines, but also for implementing core business logic in software-based organizations [9,22]. Such frameworks support software engineers in building highly scalable, reliable, and efficient applications that process continuous data streams of massive volume.…”

Benchmarking Scalability of Stream Processing Frameworks Deployed as Event-Driven Microservices in the Cloud

“…Hazelcast Jet. Hazelcast Jet [53] is a stream processing framework built on top of the Hazelcast IMDG distributed, inmemory object store. It can be embedded into Java applications and does not have any dependencies on an external system.…”

“…Instead, individual instances discover each other, form a cluster, and handle coordination and data replication internally. Hazelcast Jet differs from other frameworks in its execution model, which is based on a concept similar to coroutines and cooperative threads [53]. With the release of Hazelcast 5.0 in 2021, Hazelcast Jet has been merged with Hazelcast IMDG into one unified product.…”

“…6 However, currently this project only provides implementations for Apache Flink. Moreover, Gencer et al [53] implemented the Nexmark benchmark for their performance evaluation of Hazelcast Jet.…”