Towards a Concept for Building a Big Data Architecture with Microservices

Abstract

Microservices and Big Data are renowned hot topics in computer science that have gained a lot of hype. While the use of microservices is an approach that is used in modern software development to increase flexibility, Big Data allows organizations to turn today’s information deluge into valuable insights. Many of those Big Data architectures have rather monolithic elements. However, a new trend arises in which monolithic architectures are replaced with more modularized ones, such as microservices. This transformation provides the benefits from microservices such as modularity, evolutionary design and extensibility while maintaining the old monolithic product’s functionality. This is also valid for Big Data architectures. To facilitate the success of this transformation, there are certain beneficial factors. In this paper, those aspects will be presented and the transformation of an exemplary Big Data architecture with somewhat monolithic elements into a microservice favoured one is outlined.

Overcoming traditional ETL systems architectural problems using a service‐oriented approach

“…Despite some drawbacks related to the complexity of data flows and the many technologies involved, they perceived the resulting architecture as easier to maintain and has more fault isolation. The microservices for Big Data architecture are also explored by (Shakir et al, 2021). The extensibility behind the integration of new components and the application of best‐fitting technology for specific tasks are two examples of success factors that are compared to the proposed architecture for Big Data.…”

“…The microservices for Big Data architecture are also explored by (Shakir et al, 2021). The extensibility behind the integration of new components and the application of best-fitting technology for specific tasks are two examples of success factors that are compared to the proposed architecture for Big Data.…”

NetOr: A Microservice Oriented Inter-Domain Vertical Service Orchestrator for 5G Networks

“…Due to its decentralized nature, microservice architectures adopt a "Decentralized Governance" property. In other words, each system component can be developed using its most suitable technology, leading to higher flexibility (Shakir et al, 2021). Such property is not present in monolithic architectures where all system modules must share the same underlying code basis, leading to a lower capacity for replacing or adding new modules.…”