A Cost Model for SPARK SQL

Baldacci, Lorenzo; Golfarelli, Matteo

doi:10.1109/tkde.2018.2850339

Cited by 22 publications

(17 citation statements)

References 17 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…We say that a relational assertion P(a, b) ∈ A is redundant (based on G T ) if and only if there exist some relational assertion Q(a, b) ∈ A such that Q ≺ G T P. Since the property poset is finite, there are minimal properties that ensure that, given a redundant assertion P(a, b), we can always define a set of relational assertions from which P(a, b) can be inferred. 6 We refer to this set of non-redundant assertions from which P(a, b) can be inferred as the G T -inference base of P(a, b).…”

Section: Definitionmentioning

confidence: 99%

“…4.2.3). Consequently, for the complexity of select-project-join operations, we adapt the cost model described in [6,27] with some simplification due to the fact that we use a purely cloud-based solution.…”

Section: Complexitymentioning

confidence: 99%

“…3.2, minimal patterns computation is created by means of Query 3 that uses a groupby operator. Grounding on the analysis of [6], we can assume that its time complexity is Θ( n w ). To calculate patterns cardinality descriptors, we use Query 4 and 5 that are aggregate and groupby queries over the relation AKP SO .…”

Section: Complexitymentioning

confidence: 99%

“…It is easy to verify that the model can be easily adapted to work with equivalence classes of types and properties, rather than individual types and properties 6. It is unlikely but not impossible that an assertion P(a, b) can be inferred from more than one assertions; as an example consider the case where Q(a, b) ∈ A,R(a, b) ∈ A, Q ≺ G T P,R ≺ G T P with P and R disconnected in G T .…”

mentioning

confidence: 99%

See 3 more Smart Citations

ABSTAT-HD: a scalable tool for profiling very large knowledge graphs

et al. 2021

View full text Add to dashboard Cite

Processing large-scale and highly interconnected Knowledge Graphs (KG) is becoming crucial for many applications such as recommender systems, question answering, etc. Profiling approaches have been proposed to summarize large KGs with the aim to produce concise and meaningful representation so that they can be easily managed. However, constructing profiles and calculating several statistics such as cardinality descriptors or inferences are resource expensive. In this paper, we present ABSTAT-HD, a highly distributed profiling tool that supports users in profiling and understanding big and complex knowledge graphs. We demonstrate the impact of the new architecture of ABSTAT-HD by presenting a set of experiments that show its scalability with respect to three dimensions of the data to be processed: size, complexity and workload. The experimentation shows that our profiling framework provides informative and concise profiles, and can process and manage very large KGs.

show abstract

Section: Definitionmentioning

confidence: 99%

Section: Complexitymentioning

confidence: 99%

Section: Complexitymentioning

confidence: 99%

mentioning

confidence: 99%

See 2 more Smart Citations

ABSTAT-HD: a scalable tool for profiling very large knowledge graphs

et al. 2021

View full text Add to dashboard Cite

show abstract

“…However, time complexity was higher. In [23], a novel cost model was introduced for Spark SQL. But, the minimization of error was not enough.…”

Section: Related Workmentioning

confidence: 99%

BIBSQLQC: Brown infomax boosted SQL query clustering algorithm to detect anti-patterns in the query log

Ramakrishnan¹,

Palanisamy²

2020

Turk J Elec Eng & Comp Sci

View full text Add to dashboard Cite

Discovery of antipatterns from arbitrary SQL query log depends on the static code analysis used to enhance the quality and performance of software applications. The existence of antipatterns reduces the quality and leads to redundant SQL statements. SQL log includes a large load on the database and it is difficult for an analyst to extract large patterns in a minimal time. Existing techniques which discover antipatterns in SQL query face a lot of innumerable challenges to discover the normal sequences of queries within the log. In order to discover the antipatterns in the log, an efficient technique called Brown infomax boosted SQL query clustering (BIBSQLQC) technique is introduced. Initially, the number of patterns (i.e. queries) are extracted from the SQL query log. After extracting the patterns, the ensemble clustering process is carried out to find out the antipatterns from the given query log. The Brown infomax boost clustering is an ensemble learning method for grouping the patterns by constructing several weak learners. The Brown clustering is used as a weak learner for partitioning the patterns into 'k' number of clusters based on the Euclidean distance measure. Then the weak learner merges the two clusters with maximum information gained to minimize the time complexity. The clustering results of weak learners are combined into strong results with minimal error rate (ER). By this way, the antipattern in the SQL query log is detected with a higher accuracy. Experimental evaluation is conducted with different parameters namely detection accuracy (DA), false positive rate (FPR) and time complexity (TC) using the two SQL query log data-sets (DS). The experimental result shows that, the BIBSQLQC technique achieves higher DA with lower TC and FPR than the conventional methods.

show abstract