Intelligent container orchestration techniques for batch and micro-batch processing and data transfer.

Rovnyagin, Mikhail M.; Shipugin, Vladislav A.; Ovchinnikov, Kirill A.; Durachenko, Sergei V.

doi:10.1016/j.procs.2021.06.079

Cited by 2 publications

(5 citation statements)

References 15 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Two libraries are standard in the Spark landscape: DStream API (RDD format) and Structured Streaming (DataFrame format). The Structured Streaming has been released in the Spark 2 version, providing DataFrame API usage on the streaming data [22]. The DataFrame API is an important feature; the study focuses on that direction since the data frame is more mature and feature-rich.…”

Section: Discussionmentioning

confidence: 99%

See 1 more Smart Citation

Spark-Based Digital Factory Design

Pölöskei

2022

Acta Electrotechnica Et Informatica

View full text Add to dashboard Cite

Big data processing often uses the paradigm of parallelism by computing directly on top of the distributed data storage. The existing big data workflows unify the data processing practices to utilize the cloud’s native computational potentials to offer advanced machine learning and BI capabilities. Spark is an open-source massively parallel in-memory data processing framework, the current state-of-the-art. The primary approach is to break down the job into granular-level executed tasks, enabling parallelization. In the discussed case study, through IoT – cloud solutions, the plant data can be converted into an analyzable form to let the farther machine learning modules produce added value. To maximize the efficiency of the processing and accumulation, cloud-based components are introduced. Based on the data insights, the appropriate operative actions can be taken. The cost and performance optimization methods were also discussed in the study. Through achieving higher degree of digitalization, the control over the production increased.

show abstract

Section: Discussionmentioning

confidence: 99%

“…Next to big data, the term "fast data" should also be evaluated. Three central data processing are considered on a time basis [22]. After getting a trigger, the batch processing method does it in query time.…”

Section: Optimizationmentioning

confidence: 99%

Spark-Based Digital Factory Design

Pölöskei

2022

Acta Electrotechnica Et Informatica

View full text Add to dashboard Cite

show abstract

“…Ali and Wrembel [17] point out a key limitation in current ETL tools: the lack of efficient workflow development support due to missing automated optimization and finetuning capabilities. This gap often leads to the creation of bespoke ETL tools tailored to specific business needs [15,33,34]. The increasing data volume adds complexity to the ETL workflow design, elevating execution costs and risking operational delays or failures.…”

Section: Relevance To the Present Studymentioning

confidence: 99%

“…In batch processing, data are accumulated into groups or "batches" and processed collectively once a certain threshold or time limit is reached [34]. This approach, while an efficient and simplifying system design, can introduce latency due to the gap between data collection and processing.…”

Section: Micro-batch Data Processingmentioning

confidence: 99%

“…In stream processing, systems continuously operate on incoming data with minimal delay, maintaining an event-driven architecture. The fluctuating rate of incoming data can sometimes lead to increased computational demands, particularly during high data inflow periods [34]. While stream processing is adept at handling real-time data, integrating historical data analysis, crucial in digital twin models, may necessitate an additional system design.…”

Section: Stream Data Processingmentioning

confidence: 99%

See 1 more Smart Citation

Digital Twin Data Management: Framework and Performance Metrics of Cloud-Based ETL System

Dapkute,

Siozinys,

Jonaitis

et al. 2024

Machines

View full text Add to dashboard Cite

This study delves into the EA-SAS platform, a digital twin environment developed by our team, with a particular focus on the EA-SAS Cloud Scheduler, our bespoke program designed to optimize ETL (extract, transform, and load) scheduling and thereby enhance automation within industrial systems. We elucidate the architectural intricacies of the EA-SAS Cloud Scheduler, demonstrating its adeptness in efficiently managing computationally heavy tasks, a capability underpinned by our empirical benchmarks. The architecture of the scheduler incorporates Docker to create isolated task environments and leverages RabbitMQ for effective task distribution. Our analysis reveals the EA-SAS Cloud Scheduler’s prowess in maintaining minimal overhead times, even in scenarios characterized by high operational loads, underscoring its potential to markedly bolster operational efficiency in industrial settings. While acknowledging the limitations inherent in our current assessment, particularly in simulating real-world industrial complexities, the study also charts potential future research pathways. These include a thorough exploration of the EA-SAS Cloud Scheduler’s adaptability across diverse industrial scenarios and an examination of the integration challenges associated with its reliance on specific technological frameworks.

show abstract

Intelligent container orchestration techniques for batch and micro-batch processing and data transfer.

Cited by 2 publications

References 15 publications

Spark-Based Digital Factory Design

Spark-Based Digital Factory Design

Digital Twin Data Management: Framework and Performance Metrics of Cloud-Based ETL System

Contact Info

Product

Resources

About