Approximate Calculation of Window Aggregate Functions via Global Random Sample

Song, Guangxuan; Qu, Wenwen; Liu, Xiaojie; Wang, Xiaoling

doi:10.1007/s41019-018-0060-x

Cited by 12 publications

(5 citation statements)

References 29 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Simultaneously, there was a surge in research within the related field of Approximate Query Processing (AQP). One notable alternative to cardinality estimation and AQP problems involved sampling-based methods [16][17][18]. Unlike synopsis-based approaches like histograms, these methods required maintaining a fixed-size or fixed-proportion sample of the entire dataset, relying on this sample to provide estimated query results.…”

Section: Related Workmentioning

confidence: 99%

Cardinality estimation based on QDSPN for embedded databases under dynamic workload

Ding,

Su,

Shen

et al. 2024

Preprint

View full text Add to dashboard Cite

Cardinality estimation has been a pivotal and enduring research focus within database query optimization. While significant advancements have been made in estimating cardinalities for both individual tables and complex multi-table joins, there remains a notable gap in research pertaining to embedded database scenarios. Embedded databases are typically characterized by limited resources and a preponderance of dense, short-term hotspot queries. As a result, cardinality estimation within the constraints of embedded databases poses additional complexities and challenges. In this paper, we introduce a novel Query-driven Sum-Product Network (QDSPN), which leverages the capabilities of sum-product networks (SPNs) to learn from historical data and adapt to dynamic workload variations. This approach effectively mitigates the inherent challenges of SPNs, such as false cluster collisions and independence assumption errors, particularly under conditions of strongly correlated data. Furthermore, we propose a two-stage query clustering framework tailored for dynamic workload environments. This framework serves to guide the structural configuration of the sum-product network, enhancing its adaptability and efficiency. We conduct extensive experiments to validate the performance of QDSPN under dynamic workloads. The experimental results demonstrate the evident advantages of the proposed QDSPN, and highlight its potential for widespread adoption in embedded database systems.

show abstract

Section: Related Workmentioning

confidence: 99%

Cardinality estimation based on QDSPN for embedded databases under dynamic workload

Ding,

Su,

Shen

et al. 2024

Preprint

View full text Add to dashboard Cite

show abstract

“…Online Query Sampling techniques are widely used to support approximate query processing [84,101]. Given a time constraint in an AQP query, a sample size can be computed by estimating how many samples the system can process within the time constraint.…”

Section: Online Aggregationmentioning

confidence: 99%

“…Then, the sampling techniques can be used to get a set of samples and the selected samples are used to answer the future queries. As the distribution of many real-world datasets are proved to be uniform-like distribution and Gaussian-like distribution, early online sampling methods use random sampling to select samples [83,84,88,101]. Note that given the data-distribution assumption and random sampling can provide users with a confidence interval in most cases.…”

Section: Online Aggregationmentioning

confidence: 99%

Approximate Query Processing: What is New and Where to Go?

2018

Data Sci. Eng.

112

View full text Add to dashboard Cite

Online analytical processing (OLAP) is a core functionality in database systems. The performance of OLAP is crucial to make online decisions in many applications. However, it is rather costly to support OLAP on large datasets, especially big data, and the methods that compute exact answers cannot meet the high-performance requirement. To alleviate this problem, approximate query processing (AQP) has been proposed, which aims to find an approximate answer as close as to the exact answer efficiently. Existing AQP techniques can be broadly categorized into two categories. (1) Online aggregation: select samples online and use these samples to answer OLAP queries. (2) Offline synopses generation: generate synopses offline based on a-priori knowledge (e.g., data statistics or query workload) and use these synopses to answer OLAP queries. We discuss the research challenges in AQP and summarize existing techniques to address these challenges. In addition, we review how to use AQP to support other complex data types, e.g., spatial data and trajectory data, and support other applications, e.g., data visualization and data cleaning. We also introduce existing AQP systems and summarize their advantages and limitations. Lastly, we provide research challenges and opportunities of AQP. We believe that the survey can help the partitioners to understand existing AQP techniques and select appropriate methods in their applications.

show abstract

“…Step 1 and 2 prepare the candidate splits for step 3 to convert feature values into bin indexes. Quantile sketch is a widely-used data structure for approximate query [25,34] and is usually small in size [15,22,14], so the network overhead is almost negligible. The communication bottleneck incurs in step 4.…”

Section: Horizontal-to-vertical Transformationmentioning

confidence: 99%

An experimental evaluation of large scale GBDT systems

Jiang

Shao

et al. 2019

Proc. VLDB Endow.

View full text Add to dashboard Cite

Gradient boosting decision tree (GBDT) is a widely-used machine learning algorithm in both data analytic competitions and real-world industrial applications. Further, driven by the rapid increase in data volume, efforts have been made to train GBDT in a distributed setting to support large-scale workloads. However, we find it surprising that the existing systems manage the training dataset in different ways, but none of them have studied the impact of data management. To that end, this paper aims to study the pros and cons of different data management methods regarding the performance of distributed GBDT.We first introduce a quadrant categorization of data management policies based on data partitioning and data storage. Then we conduct an in-depth systematic analysis and summarize the advantageous scenarios of the quadrants. Based on the analysis, we further propose a novel distributed GBDT system named Vero, which adopts the unexplored composition of vertical partitioning and row-store and suits for many large-scale cases. To validate our analysis empirically, we implement different quadrants in the same code base and compare them under extensive workloads, and finally compare Vero with other state-of-the-art systems over a wide range of datasets. Our theoretical and experimental results provide a guideline on choosing a proper data management policy for a given workload.

show abstract

Approximate Calculation of Window Aggregate Functions via Global Random Sample

Cited by 12 publications

References 29 publications

Cardinality estimation based on QDSPN for embedded databases under dynamic workload

Cardinality estimation based on QDSPN for embedded databases under dynamic workload

Approximate Query Processing: What is New and Where to Go?

An experimental evaluation of large scale GBDT systems

Contact Info

Product

Resources

About