MPC with Machine Learning Applied to Resource Allocation Problem using Lambda Architecture

Pont, Mauricio P. Dal; Ferreira, Raul Sena; Teixeira, Wendell W.; Lima, Daniel Martins; Normey‐Rico, Julio E.

doi:10.1016/j.ifacol.2019.06.120

Cited by 3 publications

(2 citation statements)

References 14 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The next step is synchronization and update. The implementation of merging, combining, and synchronizing computed statistics from all three layers is defined in Equation ( 7): (7) In Equation (7), the statistics obtained from the serving layer are denoted as λ storage processed , the data retrieved from batch computations are denoted as λ processed loaded , and the data calculated from streaming data are denoted as λ stream processed . These data sets are combined (joined) and returned as a new data set λ joined [23].…”

Section: Explanation Of Coordinated Working Of Different Layers In La...mentioning

confidence: 99%

“…A design to optimally use the resources of the cluster is to use stream processing only when the batch processing time exceeds the system response time [4]. Using software performance models to predict execution metrics and cluster resources for a big-data system and then running the speed layer is an applicationspecific method [7]. Another study [8] proposed a basic set of techniques to investigate the challenges of volatility, heterogeneity, and low-latency performance aimed at reducing total system time (scheduling, execution, monitoring, and error recovery) and latent faults.…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Performance Analysis of Lambda Architecture-Based Big-Data Systems on Air/Ground Surveillance Application with ADS-B Data

Demirezen,

Navruz

2023

Sensors

View full text Add to dashboard Cite

This study introduces a novel methodology designed to assess the accuracy of data processing in the Lambda Architecture (LA), an advanced big-data framework qualified for processing streaming (data in motion) and batch (data at rest) data. Distinct from prior studies that have focused on hardware performance and scalability evaluations, our research uniquely targets the intricate aspects of data-processing accuracy within the various layers of LA. The salient contribution of this study lies in its empirical approach. For the first time, we provide empirical evidence that validates previously theoretical assertions about LA, which have remained largely unexamined due to LA’s intricate design. Our methodology encompasses the evaluation of prospective technologies across all levels of LA, the examination of layer-specific design limitations, and the implementation of a uniform software development framework across multiple layers. Specifically, our methodology employs a unique set of metrics, including data latency and processing accuracy under various conditions, which serve as critical indicators of LA’s accurate data-processing performance. Our findings compellingly illustrate LA’s “eventual consistency”. Despite potential transient inconsistencies during real-time processing in the Speed Layer (SL), the system ultimately converges to deliver precise and reliable results, as informed by the comprehensive computations of the Batch Layer (BL). This empirical validation not only confirms but also quantifies the claims posited by previous theoretical discourse, with our results indicating a 100% accuracy rate under various severe data-ingestion scenarios. We applied this methodology in a practical case study involving air/ground surveillance, a domain where data accuracy is paramount. This application demonstrates the effectiveness of the methodology using real-world data-intake scenarios, therefore distinguishing this study from hardware-centric evaluations. This study not only contributes to the existing body of knowledge on LA but also addresses a significant literature gap. By offering a novel, empirically supported methodology for testing LA, a methodology with potential applicability to other big-data architectures, this study sets a precedent for future research in this area, advancing beyond previous work that lacked empirical validation.

show abstract

Section: Explanation Of Coordinated Working Of Different Layers In La...mentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

Performance Analysis of Lambda Architecture-Based Big-Data Systems on Air/Ground Surveillance Application with ADS-B Data

Demirezen,

Navruz

2023

Sensors

View full text Add to dashboard Cite

show abstract

Application of Big Data Analysis Technology in Mechanical Design, Manufacturing and Automation

Guo

2021

The 2021 International Conference on Machine Learning and Big Data Analytics for IoT Security and Privacy

View full text Add to dashboard Cite

Approximate dynamic programming for stochastic resource allocation problems

Forootani

Iervolino

Tipaldi

et al. 2020

IEEE/CAA J. Autom. Sinica

View full text Add to dashboard Cite

A stochastic resource allocation model, based on the principles of Markov decision processes (MDPs), is proposed in this paper. In particular, a general-purpose framework is developed, which takes into account resource requests for both instant and future needs. The considered framework can handle two types of reservations (i.e., specified and unspecified time interval reservation requests), and implement an overbooking business strategy to further increase business revenues. The resulting dynamic pricing problems can be regarded as sequential decision-making problems under uncertainty, which is solved by means of stochastic dynamic programming (DP) based algorithms. In this regard, Bellman's backward principle of optimality is exploited in order to provide all the implementation mechanisms for the proposed reservation pricing algorithm. The curse of dimensionality, as the inevitable issue of the DP both for instant resource requests and future resource reservations, occurs. In particular, an approximate dynamic programming (ADP) technique based on linear function approximations is applied to solve such scalability issues. Several examples are provided to show the effectiveness of the proposed approach.

show abstract

MPC with Machine Learning Applied to Resource Allocation Problem using Lambda Architecture

Cited by 3 publications

References 14 publications

Performance Analysis of Lambda Architecture-Based Big-Data Systems on Air/Ground Surveillance Application with ADS-B Data

Performance Analysis of Lambda Architecture-Based Big-Data Systems on Air/Ground Surveillance Application with ADS-B Data

Application of Big Data Analysis Technology in Mechanical Design, Manufacturing and Automation

Approximate dynamic programming for stochastic resource allocation problems

Contact Info

Product

Resources

About