Flexible and scalable storage management for data-intensive stream processing

Botan, Irina; Alonso, Gustavo; Fischer, Peter M.; Kossmann, Donald; Tatbul, Nesime

doi:10.1145/1516360.1516467

Cited by 28 publications

(30 citation statements)

References 12 publications

Supporting

Mentioning

Contrasting

Unclassified

Order By: Relevance

“…Ingestion/storage for streaming would both benefit from detecting and then dynamically adapting to the observed stream access patterns [28], [29], ranging from fine-grained per record/tuple access to group queries (multi get/put) or scan-based.…”

Section: Missing Featuresmentioning

confidence: 99%

Towards a unified storage and ingestion architecture for stream processing

Marcu

Costan

Antoniu

et al. 2017

2017 IEEE International Conference on Big Data (Big Data)

View full text Add to dashboard Cite

Abstract-Big Data applications are rapidly moving from a batch-oriented execution model to a streaming execution model in order to extract value from the data in real-time. However, processing live data alone is often not enough: in many cases, such applications need to combine the live data with previously archived data to increase the quality of the extracted insights. Current streaming-oriented runtimes and middlewares are not flexible enough to deal with this trend, as they address ingestion (collection and pre-processing of data streams) and persistent storage (archival of intermediate results) using separate services. This separation often leads to I/O redundancy (e.g., write data twice to disk or transfer data twice over the network) and interference (e.g., I/O bottlenecks when collecting data streams and writing archival data simultaneously). In this position paper, we argue for a unified ingestion and storage architecture for streaming data that addresses the aforementioned challenge. We identify a set of constraints and benefits for such a unified model, while highlighting the important architectural aspects required to implement it in real life. Based on these aspects, we briefly sketch our plan for future work that develops the position defended in this paper.

show abstract

Section: Missing Featuresmentioning

confidence: 99%

Towards a unified storage and ingestion architecture for stream processing

Marcu

Costan

Antoniu

et al. 2017

2017 IEEE International Conference on Big Data (Big Data)

View full text Add to dashboard Cite

show abstract

“…The first generation was to develop the functionality of persistence to store the stream data, and corresponds to the studies such as [9] and [10] in this generation. On the other hand, the second generation might be the set of studies related to Cloud Computing and Big Data.…”

Section: Related Workmentioning

confidence: 99%

Analysis of Impactful Factors on Performance in Combining Architectural Elements of IoT

Kikuchi

Watanabe²,

Kondo³

et al. 2017

AIT

View full text Add to dashboard Cite

We implemented a generalized infrastructure for Internet of Things (IoT infrastructure) to be applicable in various areas such as Smart Grid. That IoT infrastructure has two methods to store sensor data. They commonly have the features of double overlay structure, virtualization of sensors, composite services as federation using publisher/subscriber. And they are implemented as synthesizing the elemental architectures. The two methods majorly have the common architectural elements, however there are differences in how to compose and utilize them. But we observed the non-negligible differences in their achieved performance by the actual implementations due to operational items beyond these architectural elements. In this paper, we present the results of our analysis about the factors of the revealed differences based on the measured performance. In particular, it is clarified that a negative side effect due to combining independent elemental micro solutions naively could be amplified, if maximizing the level of loose coupling is applied as the most prioritized design and operational policy. Primarily, these combinations should be evaluated and verified during the basic design phase. However, the variation of how to synthesize them tends to be a blind spot when adopting the multiple independent architectural elements commonly. As a practical suggestion from this case, the emphasized importance in carrying out a new synthetization with multiple architectures is to make a balance naturally among architectural elements, or solutions based on them, and there is a certain demand to establish a methodology for architectural synthetization, including verification.

show abstract

“…However, it is designed to work within the same cluster and does not address the issues of sending continuous stream of events between data-centers. In [47], the authors propose a store manager for streams, which exploits access patterns. The streams are cached in memory or disks and shared between the producers and the consumers of events; there is no support for transfers.…”

Section: Data Stream Management Systems (Dsms)mentioning

confidence: 99%