An event-based data processing system using Kafka container cluster on Kubernetes environment

Liu, Jung-Chun; Hsu, Ching‐Hsien; Zhang, Jiahao; Kristiani, Endah; Yang, Chao-Tung

doi:10.1007/s00521-023-08326-1

Cited by 3 publications

(2 citation statements)

References 35 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Literature [1] gives a new response speed optimization method of power supply system in K8S container cluster environment, but this method has great pertinency and limitations, and is difficult to be widely applied at present. Literatures [2][3][4] have studied the network transmission, load balancing, network environment and compatibility of K8S container cluster, and designed different schemes to improve performance. However, these schemes often require a lot of upgrading and optimization of the existing system, and the cost is difficult to estimate.…”

Section: Introductionmentioning

confidence: 99%

Research and Implementation of Container Based Application Orchestration Service Technology

Liu,

Wang,

Zhao

et al. 2024

J. Phys.: Conf. Ser.

View full text Add to dashboard Cite

With the rapid development of cloud computing technology, Kubernetes(K8S), as the main orchestration tool for cloud native applications, has become the preferred choice for enterprises and developers. This article is based on container based application orchestration service technology. Through a set of templates containing cloud resource descriptions, it quickly completes functions such as application creation and configuration, application batch cloning, and application multi environment deployment. It simplifies and automates the lifecycle management capabilities required for cloud applications, such as resource planning, application design, deployment, status monitoring, and scaling. Users can more conveniently complete infrastructure management and operation and maintenance work, In order to focus more on innovation and research and development, and improve work efficiency. The actual application effect of the technology used in this article depends to a certain extent on the ability level of basic service resources, and manual template creation is required for the first use. In production use, a certain professional ability is required to create a good application layout template, adjust and optimize resources according to the production environment, in order to significantly improve the effectiveness and efficiency of practical applications.

show abstract

Section: Introductionmentioning

confidence: 99%

Research and Implementation of Container Based Application Orchestration Service Technology

Liu,

Wang,

Zhao

et al. 2024

J. Phys.: Conf. Ser.

View full text Add to dashboard Cite

show abstract

“…User behavior data is synchronized to HDFS using Flume, and full and incremental service data is synchronized to HDFS using Maxwell+Flume and DataX, respectively. A Kafka cluster [6] was configured as a buffer between Flume at the collection layer and the synchronization layer to prevent the server from going down due to excessive data volume. An ETL interceptor [9] was configured on the collection tier Flume to perform simple data cleansing and filter illegal data to prevent Hive from parsing subsequent sequences.…”

Section: Introductionmentioning

confidence: 99%

A new whole process analysis framework big data based for E-commerce application

Zhou,

Hong,

et al. 2024

Proceedings of the 5th International Conference on E-Commerce and Internet Technology, ECIT 2024, March 15–17, 2024, Changsha,

View full text Add to dashboard Cite

With the rapid development of the Internet data era, the number of users of ecommerce websites has increased dramatically, and the corresponding operation data also shows a surge trend. However, many e-commerce enterprises still use traditional databases, which are difficult to effectively handle massive data. This paper proposes a full-process processing analysis system based on Hadoop by deeply analyzing the Hadoop big data ecosystem technology. Innovative use of Kafka as a buffer to prevent server crashes. A customized interceptor is used on Flume for data cleaning to avoid subsequent data warehouse parsing problems, and Spark is used to replace the underlying engine of Hive to improve computational efficiency. The whole system includes four modules: data collection, data warehouse, fully automated task scheduling and data visualization, which can effectively reduce repetitive data development of e-commerce big data enterprises in practical applications, efficiently analyze massive and real historical data, and generate visualization reports.

show abstract

Reactive-Optimized Sentence Detection In Kubernetes Using OpenNLP And Native GraalVM Image With Framework Metric Comparison

Sharma¹,

Tahiliani²,

Dubey

2023

2023 4th International Conference for Emerging Technology (INCET)

View full text Add to dashboard Cite

An event-based data processing system using Kafka container cluster on Kubernetes environment

Cited by 3 publications

References 35 publications

Research and Implementation of Container Based Application Orchestration Service Technology

Research and Implementation of Container Based Application Orchestration Service Technology

A new whole process analysis framework big data based for E-commerce application

Reactive-Optimized Sentence Detection In Kubernetes Using OpenNLP And Native GraalVM Image With Framework Metric Comparison

Contact Info

Product

Resources

About