Vendor-Agnostic Reconfiguration of Kubernetes Clusters in Cloud Federations

Abstract

Kubernetes (K8s) defines standardized APIs for container-based cluster orchestration such that it becomes possible for application managers to deploy their applications in a portable and interopable manner. However, a practical problem arises when the same application must be replicated in a distributed fashion across different edge, fog and cloud sites; namely, there will not exist a single K8s vendor that is able to provision and manage K8s clusters across all these sites. Hence, the problem of feature incompatibility between different K8s vendors arises. A large number of documented features in the open-source distribution of K8s are optional features that are turned off by default but can be activated by setting specific combinations of parameters and plug-in components in configuration manifests for the K8s control plane and worker node agents. However, none of these configuration manifests are standardized, giving K8s vendors the freedom to hide the manifests behind a single, more restricted, and proprietary customization interface. Therefore, some optional K8s features cannot be activated consistently across K8s vendors and applications that require these features cannot be run on those vendors. In this paper, we present a unified, vendor-agnostic feature management approach for consistently configuring optional K8s features across a federation of clusters hosted by different Kubernetes vendors. We describe vendor-agnostic reconfiguration tactics that are already applied in industry and that cover a wide range of optional K8s features. Based on these tactics, we design and implement an autonomic controller for declarative feature compatibility management across a cluster federation. We found that the features configured through our vendor-agnostic approach have no impact on application performance when compared with a cluster where the features are configured using the configuration manifests of the open-source K8s distribution. Moreover, the maximum time to complete reconfiguration of a single feature is within 100 seconds, which is 6 times faster than using proprietary customization interfaces of mainstream K8s vendors such as Google Kubernetes Engine. However, there is a non-negligible disruption to running applications when performing the reconfiguration to an existing cluster; this disruption impact does not appear using the proprietary customization methods of the K8s vendors due to the use of rolling upgrade of cluster nodes. Therefore, our approach is best applied in the following three use cases: (i) when starting up new K8s clusters, (ii) when optional K8s features of existing clusters must be activated as quickly as possibly and temporary disruption to running applications can be tolerated or (iii) when proprietary customization interfaces do not allow to activate the desired optional feature.

A Hierarchical Namespace Approach for Multi-Tenancy in Distributed Clouds

“…Such space isolation is achieved through OS-level namespaces [16]. In this setting, the namespace provides running environment isolation for processes, including network, file system, and process identifier (PID) space [33]. Since users can run VOLUME 12, 2024 processes in isolated namespaces, they cannot typically interfere with or monitor other processes or the host OS itself [33].…”

“…In this setting, the namespace provides running environment isolation for processes, including network, file system, and process identifier (PID) space [33]. Since users can run VOLUME 12, 2024 processes in isolated namespaces, they cannot typically interfere with or monitor other processes or the host OS itself [33]. A consequence of this isolation is the proper usage of Linuxbased security access control, known as secure computing mode (Seccomp) [34], [35].…”

“…Some isolate on the level of the container [64]. In contrast, others isolate resources within a single cluster [24], allowing tenants within the same cluster [26] to name their resources without considering other tenants [33]. AWS even allows namespaces to span regions, so the use of containment does not change even if a tenant signs in to a different AWS Region [28].…”

“…For example, it is possible for a container to delete system files using system calls or even writing on files that require privileges [71]. Linux provides a mechanism for filtering system calls -Seccomp [33], a technique widely used by modern container technologies, mobile applications, and system management services [35]. Also, many existing tools use this filtering mechanism, but it requires manual implementation of filter rules for every application [34].…”