Data Stream Management

Wingerath, Wolfram; Ritter, Norbert; Gessert, Felix

doi:10.1007/978-3-030-10555-6_4

Cited by 1 publication

(1 citation statement)

References 53 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…With the development of state-of-the-art technologies' researchers such as the M3 project proposed a disk communication layer between Reducers and Mappers to allow dynamic multi-streaming of applications and rate-based load balancing in order to achieve an efficient data processing solution (Sim et al 2018). Similarly, project Chameleon to augment the query for fast delivery used a specific contextbased indexing approach Wingerath et al [28]. Projects like Yahoo's Pig Yan et al [29], Microsoft's SCOPE Chaiken et al [30], and Google's initiatives (Hoover 2008) respectively in the context of the scholar's research are gearing to integrate declarative query constructs from the database community into MapReduce-like software to allow bigger data independence, code reusability, and instinctive query optimization.…”

Section: Current Data Processing Solutionsmentioning

confidence: 99%

Review of Knowledge Management in Optical Networks, Lambda Architecture using Database Technologies in Cloud Settings

Fofanah¹,

Koroma²,

Fofana³

2021

IJSRP

View full text Add to dashboard Cite

In this paper, we present concepts, theories, and overview of knowledge management in an autonomous optical networks and Lamba Architecture in cloud related environment. This study presents some illustrative cases that has been used to illustrate the potential application of KM architecture and to evaluate the various policies for the knowledge sharing and integration algorithm. Here the knowledge is used at the optical transponder system level, sharing, and integration implemented at the node level and supervising and data analytics (SDA) controller level. The KM process has been evaluated by integrating on a metro-network situation in terms of model error convergence time and the data shared among agents. Indeed, the propagation and reinforcement actions illustrated by similar convergence time than data-based policies at various phases of the network learning process without compromising the convergence accuracy of the model prediction. The Lambda Architecture is the new model for Big Data and database research focus, that helps in data processing with a balance on throughput, latency, and fault-tolerance. To provide a complete solution and better accuracy, low latency, and high throughput, there exists no single tool. This introduced the idea to use a set of tools and methods to build a comprehensive Big Data approach. Although this paper does not provide a developed and working tool, however, provides an outline and the methods used by researchers to overcome some of the shortcomings of Lambda Architecture. The Lambda Architecture defines a set of layers to fit in a set of tools and methods rightly for constructing a comprehensive Big Data scheme: Speed Layer, Serving Layer, Batch Layer. Each layer satisfies a set of features and builds upon the functionality delivered by the layers beneath it. The Batch Layer is the place where the master dataset is warehoused, which is an unchangeable and add-only set of raw data. Also, the batch layer computes before the results using a distributed processing system like Hadoop, Apache Spark that can manage large amounts of data. The Speed Layer encapsulates new data coming in real-time and processes it. The Serving Layer comprises a parallel processing query steam engine, that takes results from both Batch and Speed Layers and responds to questions and requests in real-time with low latency. Stack Overflow is a Question-and-Answer forum with an enormous user community, millions of posts with rapid growth over the years. This paper demonstrates the Lambda Architecture by constructing a data pipeline, to add a new "Recommended Questions" section in the Stack Overflow user profile and update the questions suggested in real-time. Additionally, various indicators such as trending tags, user performance numbers such as are shown in user dashboard by querying through batch processing layer. Finally, this paper provides a seamless search of the various methods or techniques used to help solve complex databases which are provided by Stack Overflow platform infrastructure.

show abstract

Section: Current Data Processing Solutionsmentioning

confidence: 99%