Research and implementation of HTAP for distributed database

Jing, Changhong; Liu, Wenjie; Gao, Jintao; Pei, Ouya

doi:10.1051/jnwpu/20213920430

西北工业大学学报

2021

DOI: 10.1051/jnwpu/20213920430

|View full text |Cite

Research and implementation of HTAP for distributed database

Changhong Jing

Wenjie Liu

Jintao Gao

et al.

Abstract: Data processing can be roughly divided into two categories, online transaction processing OLTP(on-line transaction processing) and online analytical processing OLAP(on-line analytical processing). OLTP is the main application of traditional relational databases, and it is some basic daily transaction processing, such as bank pipeline transactions and so on. OLAP is the main application of the data warehouse system, it supports some more complex data analysis operations, focuses on decision support, and provide… Show more

Help me understand this report

Search citation statements

Order By: Relevance

Paper Sections

Select...

Citation Types

Supporting

Mentioning

Contrasting

Year Published

2024

Publication Types

Select...

Article3

Relationship

Self Cite0

Independent3

Authors

Journals

Cited by 3 publications

References 9 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

Research and implementation of asynchronous compaction mechanism of distributed database based on LSM-Tree

DU,

LIU

2024

西北工业大学学报

View full text Add to dashboard Cite

With the continuous development of information technology, distributed database has become a research hotspot. Due to the limit support for SQL and defects in transaction processing and consistency of distributed databases based on NoSQL architecture, NewSQL databases based on LSM-Tree become gradually the mainstream of applications, such as TiDB and OceanBase. The distributed LSM-Tree storage architecture divides the data into baseline data and incremental data. Through the compaction operation, the incremental data of different partitions and the baseline data are continuously merged and stored on the disk, thereby reducing memory pressure. However, compaction will occupy a large amount of system resources and seriously affect system availability. This paper proposes an asynchronous compaction mechanism based on LSM-Tree architecture. By subdividing the compaction process, the data merging is asynchronous, which effectively shortens the time for a single compaction operation. Experiments show that the asynchronous compaction mechanism proposed in this paper can significantly shorten the data merging time and improve the robustness and usability of the system in high-frequency writing scenarios.

show abstract

Research and implementation of asynchronous compaction mechanism of distributed database based on LSM-Tree

DU,

LIU

2024

西北工业大学学报

View full text Add to dashboard Cite

show abstract

Operator parallel optimization strategy for distributed databases

LIU,

LYU

2024

JNWPU

View full text Add to dashboard Cite

With the continuous development of network technology, the scale of data has shown explosive growth, which leads gradually to replacing traditional single machine databases with distributed databases. Distributed databases solve large-scale data storage problems through collaborative work among nodes, but due to increased communication costs between nodes, its query efficiency is not as good as a standalone database. In a distributed architecture, the data of storage nodes is only used as redundancy for multiple backups, providing data recovery in case of system failure, and it is not utilized to improve query efficiency. In response to the above issues, this article proposes an operator parallel optimization strategy for distributed databases. By splitting key physical operators, the split sub requests are evenly distributed to multiple nodes in the storage layer, which are processed in parallel by multiple nodes, thereby reducing query response time. The above strategy has been applied on distributed database CBase, and experiments have shown that the parallel optimization strategy proposed in this paper can significantly shorten SQL request query time and improve system resource utilization.

show abstract

Optimization of artificial intelligence in localized big data real-time query processing task scheduling algorithm

Sun,

Sun

2024

Front. Phys.

View full text Add to dashboard Cite

IntroductionThe development of science and technology has driven rapid changes in the social environment, especially the rise of the big data environment, which has greatly increased the speed at which people obtain information. However, in the process of big data processing, the allocation of information resources is often unreasonable, leading to a decrease in efficiency. Therefore, optimizing task scheduling algorithms has become an urgent problem to be solved.MethodsThe study optimized task scheduling algorithms using artificial intelligence (AI) methods. A task scheduling algorithm optimization model was designed using support vector machine (SVM) and K-nearest neighbor (KNN) combined with fuzzy comprehensive evaluation. In this process, the performance differences of different nodes were considered to improve the rationality of resource allocation.Results and DiscussionBy comparing the task processing time before and after optimization with the total cost, the results showed that the optimized model significantly reduced task processing time and total cost. The maximum reduction in task processing time is 2935 milliseconds. In addition, the analysis of query time before and after optimization shows that the query time of the optimized model has also been reduced. The experimental results demonstrate that the proposed optimization model is practical in handling task scheduling problems and provides an effective solution for resource management in big data environments. This research not only improves the efficiency of task processing, but also provides new ideas for optimizing future scheduling algorithms.

show abstract

scite is a Brooklyn-based organization that helps researchers better discover and understand research articles through Smart Citations–citations that display the context of the citation and describe whether the article provides supporting or contrasting evidence. scite is used by students and researchers from around the world and is funded in part by the National Science Foundation and the National Institute on Drug Abuse of the National Institutes of Health.

Contact Info

customersupport@researchsolutions.com

10624 S. Eastern Ave., Ste. A-614

Henderson, NV 89052, USA

This site is protected by reCAPTCHA and the Google Privacy Policy and Terms of Service apply.

Blog Terms and Conditions API Terms Privacy Policy Contact Cookie Preferences Do Not Sell or Share My Personal Information

Made with 💙 for researchers

Part of the Research Solutions Family.

Research and implementation of HTAP for distributed database

Cited by 3 publications

References 9 publications

Research and implementation of asynchronous compaction mechanism of distributed database based on LSM-Tree

Research and implementation of asynchronous compaction mechanism of distributed database based on LSM-Tree

Operator parallel optimization strategy for distributed databases

Optimization of artificial intelligence in localized big data real-time query processing task scheduling algorithm

Contact Info

Product

Resources

About