Optimizing Performance and Resource Consumption of Cloud-Native Logging Application Stacks

Abstract

Nowadays cloud-based applications and Internet of Things use-cases are becoming more and more common in the field of IT benefiting from the virtually limitless resources and microservice-based deployment options available in the cloud. Observability in such environments is key for tracing application execution to detect possible malfunctions and anomalies. Collecting logs can greatly help in this regard, however, a high volume of logging data can add huge costs for the maintenance of the infrastructure gathering monitoring data. In order to increase the profitability of the application, monitoring-related infrastructure needs to have the lowest cost possible while still being able to fully serve the application's monitoring needs. In this work, we investigate this aspect and provide an evaluation of the resource footprint of one of the most prominent log collection services, Elastic Stack, from the perspective of its write path.