Smart Data Placement Using Storage-as-a-Service Model for Big Data Pipelines

Abstract

Big data pipelines are developed to process data characterized by one or more of the three big data features, commonly known as the three Vs (volume, velocity, and variety), through a series of steps (e.g., extract, transform, and move), making the ground work for the use of advanced analytics and ML/AI techniques. Computing continuum (i.e., cloud/fog/edge) allows access to virtually infinite amount of resources, where data pipelines could be executed at scale; however, the implementation of data pipelines on the continuum is a complex task that needs to take computing resources, data transmission channels, triggers, data transfer methods, integration of message queues, etc., into account. The task becomes even more challenging when data storage is considered as part of the data pipelines. Local storage is expensive, hard to maintain, and comes with several challenges (e.g., data availability, data security, and backup). The use of cloud storage, i.e., storage-as-a-service (StaaS), instead of local storage has the potential of providing more flexibility in terms of scalability, fault tolerance, and availability. In this article, we propose a generic approach to integrate StaaS with data pipelines, i.e., computation on an on-premise server or on a specific cloud, but integration with StaaS, and develop a ranking method for available storage options based on five key parameters: cost, proximity, network performance, server-side encryption, and user weights/preferences. The evaluation carried out demonstrates the effectiveness of the proposed approach in terms of data transfer performance, utility of the individual parameters, and feasibility of dynamic selection of a storage option based on four primary user scenarios.

Predicting Frequent-Infrequent Access of Objects in Tiered Cloud Storage Services

“…In addressing the complexities of multi-cloud optimization, the study conducted by Quddus et al [15] presents a comprehensive framework aimed at minimizing operational costs within online social networks, leveraging dynamic data replication and strategic placement algorithms within a dual-level multi-cloud infrastructure. This research meticulously examines the multifaceted challenges inherent in multi-cloud environments, including vendor lock-in, service availability discrepancies, suboptimal cost utilization, and the exacerbation of latency during data replication processes across diverse cloud service platforms.…”

“…The paper [15] explores a novel approach to optimize cloud storage costs by utilizing a rule-based classification system. This system classifies data across four distinct storage tiers-premium, hot, cold, and archive-based on factors such as access frequency, size, and age of data.…”

Smart Data Placement for Big Data Pipelines: An Approach based on the Storage-as-a-Service Model

“…For this very evaluation model, four different parameters are selected in addition to the user weights. These parameters are as follows: cost (i.e., based on storage, bandwidth, and READ and WRITE operations -see [3]), proximity (i.e., using IP ranges provided by the cloud service providers and GeoIP), network performance (i.e., throughput), the impact of server-side encryption (i.e., performance). Figure 3 shows the evaluation matrix for the proposed method.…”

“…constraints. To this end, we first propose an approach to realize big data pipelines with hybrid infrastructure, i.e., computation on an on-premise server or on a specific cloud, but integration with StaaS; and secondly develop a ranking method to find the most suitable storage facility dynamically based on the user's requirements, including cost, proximity, network performance, impact of server-side encryption, and user weights [3].…”

Big Data Pipeline Infrastructure Design in Msme E-Commerce Systems With a Focus on Data Source Processing Using Orchestration Tools

“…Big Data Pipeline is a series of steps and tools used to collect, transform and input data from various sources into a system that can be accessed and processed [4]. In the scope of performance on the MSME e-commerce platform, Big Data Pipeline plays a role in overcoming obstacles in organizing large data, converting it into a more structured format, and storing it in an appropriate system [5]. So that it can make more targeted decisions and based on accurate data analysis.…”