Kafka: the Database Inverted, but Not Garbled or Compromised

Rooney, Seán; Urbanetz, Peter; Giblin, Chris; Bauer, Daniel; Froese, Florian; Garcés-Erice, Luis; Tomić, Saša

doi:10.1109/bigdata47090.2019.9005583

Cited by 7 publications

(4 citation statements)

References 8 publications

Supporting

Mentioning

Contrasting

Unclassified

Order By: Relevance

“…In recent years, there have been some researches on the performance of Kafka messaging system in the field of distributed messaging system [14,15,16,29,30]. For instance, in order to set the configuration of the Kafka system correctly under certain hardware conditions to ensure its performance, Han et al [29] analyzed the structure and workflow of Kafka and proposed a queue-based package flow model to predict the performance of Kafka cloud services.…”

Section: Related Workmentioning

confidence: 99%

“…In addition, Sean et al [15] wanted to find the practical problems that arise when companies use Kafka as a single data store, and to be able to propose solutions to solve these problems. To this end, they proposed some preliminary approaches to ensure the consistency of data from multiple database tables when distributed over Kafka, and how to solve compliance problems by encrypting/decrypting data from Kafka producers and consumers.…”

Section: Related Workmentioning

confidence: 99%

See 1 more Smart Citation

Formalization and verification of Kafka messaging mechanism using CSP

Yin

Zhu

et al. 2023

ComSIS

View full text Add to dashboard Cite

Apache Kafka is an open source distributed messaging system based on the publish-subscribe model, which achieves low latency, high throughput and good load balancing. As a popular messaging system, the transmission of messages between applications is one of the core functions of Kafka. Therefore, the reliability and security of data in the process of message transmission in Kafka have become the focus of attention. The formal methods can analyze whether a model is highly credible. Therefore, it is significant to analyze Kafka messaging mechanism which describes the communication process and rules between each module entity in Kafka from the perspective of formal methods. In this paper, we apply the process algebra CSP (Communicating Sequential Processes) and the model checking tool PAT (Process Analysis Toolkit) to analyze Kafka messaging mechanism. The results of verification show that the model caters for its specification and guarantees the reliability of messages in the normal communication process. Moreover, in order to further analyze the security of Kafka messaging mechanism, we add the intruder model and the authentication protocol Kerberos model and compare the verification results of Kafka messaging mechanism with or without the secure protocol Kerberos. The results show that the Kerberos protocol has improved the security of Kafka messaging mechanism in some aspects, but there are still some security loopholes.

show abstract

Section: Related Workmentioning

confidence: 99%

Section: Related Workmentioning

confidence: 99%

Formalization and verification of Kafka messaging mechanism using CSP

Yin

Zhu

et al. 2023

ComSIS

View full text Add to dashboard Cite

show abstract

“…Se ha demostrado que los lagos de datos permiten a las organizaciones procesar datos a una escala que antes no hubiera sido factible sin sistemas especializados y costosos. Facilitando a las organizaciones almacenar datos sin procesos complejos de ETL, pero sí asociados a un conjunto de políticas, para ser procesado por múltiples equipos diferentes y de acuerdo con sus necesidades (Rooney et al, 2019).…”

Section: Repositorios De Información De Salud -Lagos De Datosunclassified

Retos y oportunidades en la accesibilidad de datos

Calva

2022

Interfases

View full text Add to dashboard Cite

En el área de la salud donde existe una gran variedad y cantidad de información sensible y valiosa no es explotada en su totalidad en beneficio de las personas debido a la falta de acceso a los datos. Hay varios factores que influyen en la accesibilidad de los datos a nivel de organizaciones privadas o públicas, para determinar estos factores se utilizó una revisión bibliográfica, de los 300 artículos consultados, 23 se incluyeron en esta revisión. Entre los factores resultantes que influyen en la accesibilidad de la información se destacaron: gobiernos abiertos, sistemas de información de salud, leyes de protección de datos personales, ética en el uso de la información médica y lagos de datos. Facilitar la accesibilidad a los datos de salud derivaría en una mejora de servicios y planes de tratamientos, ahorro de los estados en recursos económicos, además impulsa la educación y la investigación.

show abstract

“…Producer and consumer can be written in different languages like Java, Python or Scala etc. There are also producers already defined (owners) such as in Azure [52], CDC (Change Data Capture) techniques [53] which make it possible to detect changes (insert, update, etc.) on the rela- • Simple and easy to implement as per BI system concerns • The basic methodology has not changed.…”

Section: Data Streams Collect and Ingestionmentioning

confidence: 99%

Data Stream Summary in Big Data Context: Challenges and Opportunities

Sarr¹,

Boly²,

Bame³

2021

Adv. sci. technol. eng. syst. j.

View full text Add to dashboard Cite

With the advent of Big Data, we are witnessing a rapid and varied production of huge amounts of sequential data that can have multiple dimensions, we speak of data streams. The characteristics of these data streams make their processing and storage very difficult and at the same time reduce the possibilities of querying them a posteriori. Thus, it has become necessary to set up so-called summary structures, equivalent to views on the data streams which take into account these constraints and allow querying the data already pruned from the system. In order to take into account the aspect of volume, speed and variety of data streams, new methods have appeared in the field of Big Data and NoSQL. These solutions combined make it possible now to set up summaries that make it possible to store and process different types of data streams with more efficiency and representativeness and which best meet the constraints of memory and CPU resources necessary for processing data streams but also with some limits.

show abstract

Kafka: the Database Inverted, but Not Garbled or Compromised

Cited by 7 publications

References 8 publications

Formalization and verification of Kafka messaging mechanism using CSP

Formalization and verification of Kafka messaging mechanism using CSP

Retos y oportunidades en la accesibilidad de datos

Data Stream Summary in Big Data Context: Challenges and Opportunities

Contact Info

Product

Resources

About