Performance Comparison between S3, HDFS and RDS storage technologies for real-time big-data applications

Jamal, Alaa; Fleiner, Rita; Kail, Eszter

doi:10.1109/saci51354.2021.9465594

Cited by 2 publications

(1 citation statement)

References 6 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Conducting performance analyses related to database migrations within AWS ecosystems provides invaluable insights into optimizing database infrastructure to meet evolving demands. Similarly, studies focusing on storage technologies offer guidance in selecting suitable solutions for real-time big data applications, thereby enhancing the efficiency and effectiveness of data processing within AWS environments [5], [6]. Furthermore, research efforts directed towards document retrieval techniques and comparative evaluations of distributed database systems contribute significantly to the development and optimization of applications deployed on AWS.…”

Section: Literature Reviewmentioning

confidence: 99%

Database Version Control: AWS and RDS Integration with Python

2024

IJFMR

View full text Add to dashboard Cite

This paper presents an automated approach for managing database versioning and upgrades through the integration of Amazon Web Services Relational Database Service (AWS RDS) metadata into MongoDB using Python. By leveraging this integration, the system streamlines the process of version control and upgrade planning, improving efficiency and reliability compared to manual methods. Through experiments, we demonstrate the effectiveness and scalability of our approach, offering a practical solution to enhance database management workflows.

show abstract

Section: Literature Reviewmentioning

confidence: 99%

Database Version Control: AWS and RDS Integration with Python

2024

IJFMR

View full text Add to dashboard Cite

show abstract

Analyzing the Performance of the S3 Object Storage API for HPC Workloads

Gadban

Kunkel

2021

Applied Sciences

View full text Add to dashboard Cite

The line between HPC and Cloud is getting blurry: Performance is still the main driver in HPC, while cloud storage systems are assumed to offer low latency, high throughput, high availability, and scalability. The Simple Storage Service S3 has emerged as the de facto storage API for object storage in the Cloud. This paper seeks to check if the S3 API is already a viable alternative for HPC access patterns in terms of performance or if further performance advancements are necessary. For this purpose: (a) We extend two common HPC I/O benchmarks—the IO500 and MD-Workbench—to quantify the performance of the S3 API. We perform the analysis on the Mistral supercomputer by launching the enhanced benchmarks against different S3 implementations: on-premises (Swift, MinIO) and in the Cloud (Google, IBM…). We find that these implementations do not yet meet the demanding performance and scalability expectations of HPC workloads. (b) We aim to identify the cause for the performance loss by systematically replacing parts of a popular S3 client library with lightweight replacements of lower stack components. The created S3Embedded library is highly scalable and leverages the shared cluster file systems of HPC infrastructure to accommodate arbitrary S3 client applications. Another introduced library, S3remote, uses TCP/IP for communication instead of HTTP; it provides a single local S3 gateway on each node. By broadening the scope of the IO500, this research enables the community to track the performance growth of S3 and encourage sharing best practices for performance optimization. The analysis also proves that there can be a performance convergence—at the storage level—between Cloud and HPC over time by using a high-performance S3 library like S3Embedded.

show abstract

Performance Comparison between S3, HDFS and RDS storage technologies for real-time big-data applications

Cited by 2 publications

References 6 publications

Database Version Control: AWS and RDS Integration with Python

Database Version Control: AWS and RDS Integration with Python

Analyzing the Performance of the S3 Object Storage API for HPC Workloads

Contact Info

Product

Resources

About