Mission possible: Unify HPC and Big Data stacks towards application-defined blobs at the storage layer

This work aims to identify techniques leading to a highly available request processing service by using the natural decentralization and the dispersion power of the hash function involved in a Distributed Hash Table (DHT). High availability is present mainly in systems that: scale well, are balanced and are fault tolerant. These are essential features of the Distributed Hash Tables (DHTs), which have been used mainly for storage purposes. The novelty of this paper’s approach is essentially based on hash functions and decentralized Distributed Hash Tables (DHTs), which lead to highly available data solutions, which a main building block to obtain an improved platform that offers high availability for processing clients’ requests. It is achieved by using a database constructed also on a DHT, which gives high availability to its data. Further, the model requires no changes in the interface, that the request processing service already has towards its clients. Subsequently, the DHT layer is added, for the service to run on top of it, and also a load balancing front end, in order to make it highly available, towards its clients. The paper shows, via experimental validation, the good qualities of the new request processing service, by arguing its improved scalability, load balancing and fault tolerance model.

show abstract

“…Storing the data in distributed filesystems [24, 25,32], in classical SQL databases [22] and more recently in No-SQL distributed databases [8];…”

Section: The Aspect Of Storing Data In Decentralized Systemsmentioning

confidence: 99%

Towards a Highly Available Model for Processing Service Requests Based on Distributed Hash Tables

Iancu

Țăpuș

2022

Mathematics

View full text Add to dashboard Cite

show abstract

“…Given the workflow nature of many state-of-the-art simulators for scientific computing, Srirama et al [123] proposed a workflow-partitioning strategy to reduce the data communication in the resulting deployment. Matri et al [124], [125] analyzed the applicability of binary large objects (known as blobs) and object storage systems to solve the problems with POSIX-IO-compliant file systems and as a mechanism to replace distributed file systems for BDA analytics.…”

Section: ) Infrastructure: Distributed Storage and Cloud Computingmentioning

confidence: 99%

Toward High-Performance Computing and Big Data Analytics Convergence: The Case of Spark-DIY

et al. 2019

View full text Add to dashboard Cite

Convergence between high-performance computing (HPC) and big data analytics (BDA) is currently an established research area that has spawned new opportunities for unifying the platform layer and data abstractions in these ecosystems. This work presents an architectural model that enables the interoperability of established BDA and HPC execution models, reflecting the key design features that interest both the HPC and BDA communities, and including an abstract data collection and operational model that generates a unified interface for hybrid applications. This architecture can be implemented in different ways depending on the process-and data-centric platforms of choice and the mechanisms put in place to effectively meet the requirements of the architecture. The Spark-DIY platform is introduced in the paper as a prototype implementation of the architecture proposed. It preserves the interfaces and execution environment of the popular BDA platform Apache Spark, making it compatible with any Spark-based application and tool, while providing efficient communication and kernel execution via DIY, a powerful communication pattern library built on top of MPI. Later, Spark-DIY is analyzed in terms of performance by building a representative use case from the hydrogeology domain, EnKF-HGS. This application is a clear example of how current HPC simulations are evolving toward hybrid HPC-BDA applications, integrating HPC simulations within a BDA environment. INDEX TERMS Big data analytics, high performance computing, spark, DIY, MPI.

show abstract

“…is study rst understands big data from the theoretical level. American scholars have summarized that big data has the following ve characteristics: huge volume, high speed, diversity, low value density, and authenticity [1]. Big data is a new resource and tool for everyone.…”

Section: Introductionmentioning

confidence: 99%

Optimization of the Economic and Trade Management Legal Model Based on the Support Vector Machine Algorithm and Logistic Regression Algorithm

Nie

Bai

Nie

et al. 2022

Mathematical Problems in Engineering

View full text Add to dashboard Cite

Nowadays, various algorithms are widely used in the field of economy and trade, and economic and trade management laws also need to introduce scientific and effective data models for optimization. In this paper, support vector machine algorithm and logistic regression algorithm are used to analyze and process the actual economic and trade case data and bank loan user data, and a hybrid model of support vector machine and logistic regression is established. This study first introduces the basic definitions and contents of the support vector machine algorithm and logistic regression algorithm, and then constructs a hybrid model by randomly dividing the data, first using the support vector machine algorithm to calculate the results, and then inputting them into the logistic regression algorithm. The first mock exam is that the efficiency of the hybrid model is much higher than that of the single model. This study mainly optimizes and upgrades the legal system of economic and trade management from two aspects. In the prediction of economic and trade legal cases, the hybrid model is significantly better than FastText and LSTM models in accuracy and macro recall performance. In terms of credit risk prediction of economic and trade loan users, the subset most likely to default in the loan customer set is obtained.

show abstract

Mission possible: Unify HPC and Big Data stacks towards application-defined blobs at the storage layer

Cited by 5 publications

References 25 publications

Towards a Highly Available Model for Processing Service Requests Based on Distributed Hash Tables

Towards a Highly Available Model for Processing Service Requests Based on Distributed Hash Tables

Toward High-Performance Computing and Big Data Analytics Convergence: The Case of Spark-DIY

Optimization of the Economic and Trade Management Legal Model Based on the Support Vector Machine Algorithm and Logistic Regression Algorithm

Contact Info

Product

Resources

About