RIDE: real-time massive image processing platform on distributed environment

Kim, Yoon Ki; Kim, Yongsung; Jeong, Cherlhyun

doi:10.1186/s13640-018-0279-5

Cited by 12 publications

(7 citation statements)

References 19 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Based on them, such event-driven products are built as: IoT [1][2], which work with the flow of messages from "smart" things; web platforms [3] that display the results of receiving or processing events to the end user; data processing pipelines [4][5][6][7], responding to events of different nature, and others. The peculiarity of such systems is that they focus not so much on reducing the processing time of each individual event and bringing it closer to real time, but on the guaranteed and clear sequence of interconnected events and guarantees the processing of each of them.…”

Section: Introductionmentioning

confidence: 99%

The Mechanism for Creation of Event-Driven Applications Based on Wolfram Mathematica and Apache Kafka

Zolotariov¹

2021

ITSSI

View full text Add to dashboard Cite

The article is devoted to the research and development of the mechanism of interaction between Wolfram Mathematica programs and Apache Kafka queue to provide the ability to build event-driven applications based on it. The subject of the research is the practical principles of building a mechanism for interaction between Wolfram Mathematica and Apache Kafka. The purpose of the article is to develop and substantiate practical recommendations regarding the formation of a mechanism for publishing messages to the Apache Kafka queue and reading messages from it for programs of the mathematical processor Wolfram Mathematica, which will make it possible to build event-driven applications. Tasks: to determine the mechanism of such interaction, prove the choice of tools for its implementation, create and test the obtained results. The research used the following tools: Apache Kafka, Kafkacat, the method of developing the Wolfram Mathematica package. The results of the research: the mechanism of interaction between Wolfram Mathematica and Apache Kafka was determined and the corresponding toolkit was created on its basis in the form of two Mathematica packages, which are built on using Apache Kafka as a queue client and third-party Kafkacat software, respectively. It is shown that the first option is less reliable and consumes much more computer resources during operation. It has been demonstrated that the Mathematica processor is currently not suitable in its pure form for real-time data analysis. Recommendations are given regarding the use of built-in compilation functions to increase the speed of such processing. Conclusions. Practical recommendations have been developed and substantiated regarding the formation of the mechanism of interaction between the Wolfram Mathematica mathematical processor and the Apache Kafka queue manager for the possibility of working in two directions with the queue: publishing messages and reading them. A toolkit for such interaction in the form of Mathematica packages has been created, their capabilities have been demonstrated, as well as comparison with each other. The economic benefit of using the described tools is shown. Future ways of its improvement are given.

show abstract

Section: Introductionmentioning

confidence: 99%

The Mechanism for Creation of Event-Driven Applications Based on Wolfram Mathematica and Apache Kafka

Zolotariov¹

2021

ITSSI

View full text Add to dashboard Cite

show abstract

“…Therefore, bottleneck problems caused by scheduling can be avoided and the communication between nodes can be decreased, reducing the network load. Because of these properties, Kafka can be a suitable framework in a real-time environment, and it was validated in [21].…”

Section: Framework For Real-time Distributed Processingmentioning

confidence: 99%

Real-time car tracking system based on surveillance videos

Jung

Kim²,

Hwang

2018

J Image Video Proc.

Self Cite

View full text Add to dashboard Cite

As a variety of video surveillance devices such as CCTV, drones, and car dashboard cameras have become popular, numerous studies have been conducted regarding the effective enforcement of security and surveillance based on video analysis. In particular, in car-related surveillance, car tracking is the most challenging task. One early approach to accomplish such a task was to analyze frames from different video sources separately. Considering the shooting range of the bulk of video devices, the outcome from the analysis of single video source is highly limited. To obtain more comprehensive information for car tacking, a set of video sources should be considered together and the relevant information should be integrated according to spatial and temporal constraints. Therefore, in this study, we propose a real-time car tracking system based on surveillance videos from diverse devices including CCTV, dashboard cameras, and drones. For scalability and fault tolerance, our system is built on a distributed processing framework and comprises a Frame Distributor, a Feature Extractor, and an Information Manager. The Frame Distributor is responsible for distributing the video frames from various devices to the processing nodes. The Feature Extractor extracts principal vehicle features such as plate number, location, and time from each frame. The Information Manager stores all the features into a database and handles user requests by collecting relevant information from the feature database. To illustrate the effectiveness of our proposed system, we implemented a prototype system and performed a number of experiments. We report some of the results.

show abstract

“…Recent years have been marked by the rapid development of software products built on microservice architecture: web services [1][2], Internet banking [3], data streaming [4][5], Internet of Things (IoT) [6][7] and others. Its advantages include: ease of construction, updating and modification of elements (microservices), ease of addition, removal and replacement due to the independence of operationas a consequence, much greater flexibility in the development of the system as a whole.…”

Section: Introductionmentioning

confidence: 99%

Microservice Architecture for Building High-Availability Distributed Automated Computing System in a Cloud Infrastructure

Zolotariov¹

2021

ITSSI

View full text Add to dashboard Cite

The article is devoted to the research and development of a highly available distributed automated computing system by iterative algorithms based on the microservice architecture in a cloud infrastructure. The subject of the research is the practical foundations of building high-availability automated computing systems based on microservice architecture in a cloud-based distributed infrastructure. The purpose of the article is to develop and to substantiate practical recommendations for the formation of the infrastructure of a high-availability automated computing system based on the microservice architecture, the choice of its constituent elements and their components. The task of the work: to identify the necessary structural elements of a microservice automated computing system and to analyze the constituent components and functional load for each of them, set specific tasks for building each of them and justify the choice of tools for their creation. In the course of the research, methods of system analysis were used to decompose a complex system into elements and each element into functional components, and tools: information technologies Apache Kafka, Kafkacat, Wolfram Mathematica, nginx, Lumen, Telegram, Dropbox, and MySQL. As a result of the study, it was found that the system infrastructure should consist of: fault-tolerant interservice transport, a high-availability computing microservice, and communication microservices with end customers, which save or process the results. For each of them, recommendations are provided regarding the formation and selection of implementation tools. According to the recommendations, one variant of implementation of such system has been developed, the principles of its operation are shown and the results are presented. It has been proven that when using a Kafka queue it is efficient to publish batches of results rather than one at a time, which results to significant overhead on queue servers and data latency for its clients. Recommendations are given on the implementation of the CI/CD system to build a continuous cycle of adding and improving microservices. Conclusions. Practical foundations have been developed for the implementation of high availability distributed automated computing systems based on microservice architecture in a cloud infrastructure. The flexibility in processing the results of such a system is shown due to the possibility of adding microservices and using third-party analytical applications that support connection to the Kafka queue. The economic benefit of using the described system is shown. Future ways of its improvement are given.

show abstract

RIDE: real-time massive image processing platform on distributed environment

Cited by 12 publications

References 19 publications

The Mechanism for Creation of Event-Driven Applications Based on Wolfram Mathematica and Apache Kafka

The Mechanism for Creation of Event-Driven Applications Based on Wolfram Mathematica and Apache Kafka

Real-time car tracking system based on surveillance videos

Microservice Architecture for Building High-Availability Distributed Automated Computing System in a Cloud Infrastructure

Contact Info

Product

Resources

About