The Distributed System of Automated Computing Based on Cloud Infrastructure

Zolotariov, Denis

doi:10.30837/itssi.2020.14.047

Cited by 5 publications

(19 citation statements)

References 10 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…As a platform for the deployment of the queue, cloud technologies DigitalOcean were chosen, which is one of the world leaders, where servers based on Ubuntu 20.04 LTS x64 OS were located, which proved to be a reliable and fast platform in previous developments [20].…”

Section: Work Demonstration and Comparisonmentioning

confidence: 99%

The Mechanism for Creation of Event-Driven Applications Based on Wolfram Mathematica and Apache Kafka

Zolotariov¹

2021

ITSSI

Self Cite

View full text Add to dashboard Cite

The article is devoted to the research and development of the mechanism of interaction between Wolfram Mathematica programs and Apache Kafka queue to provide the ability to build event-driven applications based on it. The subject of the research is the practical principles of building a mechanism for interaction between Wolfram Mathematica and Apache Kafka. The purpose of the article is to develop and substantiate practical recommendations regarding the formation of a mechanism for publishing messages to the Apache Kafka queue and reading messages from it for programs of the mathematical processor Wolfram Mathematica, which will make it possible to build event-driven applications. Tasks: to determine the mechanism of such interaction, prove the choice of tools for its implementation, create and test the obtained results. The research used the following tools: Apache Kafka, Kafkacat, the method of developing the Wolfram Mathematica package. The results of the research: the mechanism of interaction between Wolfram Mathematica and Apache Kafka was determined and the corresponding toolkit was created on its basis in the form of two Mathematica packages, which are built on using Apache Kafka as a queue client and third-party Kafkacat software, respectively. It is shown that the first option is less reliable and consumes much more computer resources during operation. It has been demonstrated that the Mathematica processor is currently not suitable in its pure form for real-time data analysis. Recommendations are given regarding the use of built-in compilation functions to increase the speed of such processing. Conclusions. Practical recommendations have been developed and substantiated regarding the formation of the mechanism of interaction between the Wolfram Mathematica mathematical processor and the Apache Kafka queue manager for the possibility of working in two directions with the queue: publishing messages and reading them. A toolkit for such interaction in the form of Mathematica packages has been created, their capabilities have been demonstrated, as well as comparison with each other. The economic benefit of using the described tools is shown. Future ways of its improvement are given.

show abstract

Section: Work Demonstration and Comparisonmentioning

confidence: 99%

The Mechanism for Creation of Event-Driven Applications Based on Wolfram Mathematica and Apache Kafka

Zolotariov¹

2021

ITSSI

Self Cite

View full text Add to dashboard Cite

show abstract

“…Third, long-term storage of computational results in a random access system. And, fourth, maintaining backward compatibility with the customer system developed in [8].…”

Section: General Structure Of the Systemmentioning

confidence: 99%

“…Microservice, which is responsible for maintaining backward compatibility with the system described in [8],receives the results of calculations of each iteration from the transport, processes them for storage in separate files in the format of the specified system, saves and sends them to Dropbox. Also, the system can have many other microservices, which is implied in the architectural approach.…”

Section: General Structure Of the Systemmentioning

confidence: 99%

“…Therefore, it is not surprising that the advantages and prospects of such architecture are appreciated by more and more developers of complex services, and gradually bring its elements to non-profit areas. As shown in [8], the achievements of modern applied science are largely the result of interdisciplinary teams of scientists, researchers and engineers, resulting in the use of each member of the group familiar in its field tools for development, research and analysis that can be combined only through specialized interaction systems for the transfer of results between participants. Also, such systems that allow the exchange of research results in automatic mode in real or near real time, have become relevant in today's environment, when researchers may be distributed around the world or otherwise not be able to be with the team.…”

Section: Introductionmentioning

confidence: 99%

“…This work is a logical continuation and extension of the ideas presented for the first time in [8]. This article focuses on the problem of building a system of automated calculations based on microservice architecture for the use of iterative algorithms based on cloud IaaS service, and is designed to use the widest range of both service microservices and end customers through the use of queue manager as a communication channel.…”

Section: Introductionmentioning

confidence: 99%

See 2 more Smart Citations

Microservice Architecture for Building High-Availability Distributed Automated Computing System in a Cloud Infrastructure

Zolotariov¹

2021

ITSSI

Self Cite

View full text Add to dashboard Cite

The article is devoted to the research and development of a highly available distributed automated computing system by iterative algorithms based on the microservice architecture in a cloud infrastructure. The subject of the research is the practical foundations of building high-availability automated computing systems based on microservice architecture in a cloud-based distributed infrastructure. The purpose of the article is to develop and to substantiate practical recommendations for the formation of the infrastructure of a high-availability automated computing system based on the microservice architecture, the choice of its constituent elements and their components. The task of the work: to identify the necessary structural elements of a microservice automated computing system and to analyze the constituent components and functional load for each of them, set specific tasks for building each of them and justify the choice of tools for their creation. In the course of the research, methods of system analysis were used to decompose a complex system into elements and each element into functional components, and tools: information technologies Apache Kafka, Kafkacat, Wolfram Mathematica, nginx, Lumen, Telegram, Dropbox, and MySQL. As a result of the study, it was found that the system infrastructure should consist of: fault-tolerant interservice transport, a high-availability computing microservice, and communication microservices with end customers, which save or process the results. For each of them, recommendations are provided regarding the formation and selection of implementation tools. According to the recommendations, one variant of implementation of such system has been developed, the principles of its operation are shown and the results are presented. It has been proven that when using a Kafka queue it is efficient to publish batches of results rather than one at a time, which results to significant overhead on queue servers and data latency for its clients. Recommendations are given on the implementation of the CI/CD system to build a continuous cycle of adding and improving microservices. Conclusions. Practical foundations have been developed for the implementation of high availability distributed automated computing systems based on microservice architecture in a cloud infrastructure. The flexibility in processing the results of such a system is shown due to the possibility of adding microservices and using third-party analytical applications that support connection to the Kafka queue. The economic benefit of using the described system is shown. Future ways of its improvement are given.

show abstract

Application of Distributed High-Precision Data Acquisition System Based on GPRS Wireless Network

Fengling

2021

Lecture Notes on Data Engineering and Communications Technologies

View full text Add to dashboard Cite

The Distributed System of Automated Computing Based on Cloud Infrastructure

Cited by 5 publications

References 10 publications

The Mechanism for Creation of Event-Driven Applications Based on Wolfram Mathematica and Apache Kafka

The Mechanism for Creation of Event-Driven Applications Based on Wolfram Mathematica and Apache Kafka

Microservice Architecture for Building High-Availability Distributed Automated Computing System in a Cloud Infrastructure

Application of Distributed High-Precision Data Acquisition System Based on GPRS Wireless Network

Contact Info

Product

Resources

About