QALD-9-plus: A Multilingual Dataset for Question Answering over DBpedia and Wikidata Translated by Native Speakers

Perevalov, Aleksandr; Diefenbach, Dennis; Usbeck, Ricardo; Both, Andreas

doi:10.1109/icsc52841.2022.00045

Cited by 26 publications

(19 citation statements)

References 0 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…In this section, we present the experimental evaluation conducted on three extensively utilized KBQA datasets, namely LC-QuAD 2.0 [18], QALD-9 plus [19], and QALD-10 [20], to assess the viability and efficacy of our proposed TSET model. We begin by introducing the experimental settings, encompassing the datasets used, the evaluation metric employed, and the implementation details.…”

Section: Methodsmentioning

confidence: 99%

“…Experimental results show that our model can significantly improve the quality of SPARQL query generation. On three well-known KBQA datasets, LC-QuAD 2.0 [18], QALD-9 plus [19], and QALD-10 [20], TSET surpasses all previous methods in answer F1 score and Query Match (QM) accuracy, achieving new state-of-the-art performance. We also do a comprehensive set of ablation studies to demonstrate the effectiveness of our proposed Triple Structure Correction (TSC) objective and the semantic transformation approach.…”

Section: Introductionmentioning

confidence: 94%

“…In this work, we utilize three widely used datasets, all based on Wikidata: LC-QuAD 2.0 [18], QALD-9 plus [19], and QALD-10 [20].…”

Section: Datasetsmentioning

confidence: 99%

See 2 more Smart Citations

Enhancing SPARQL Query Generation for Knowledge Base Question Answering Systems by Learning to Correct Triplets

Qi,

Su,

Guo

et al. 2024

Applied Sciences

View full text Add to dashboard Cite

Generating SPARQL queries from natural language questions is challenging in Knowledge Base Question Answering (KBQA) systems. The current state-of-the-art models heavily rely on fine-tuning pretrained models such as T5. However, these methods still encounter critical issues such as triple-flip errors (e.g., (subject, relation, object) is predicted as (object, relation, subject)). To address this limitation, we introduce TSET (Triplet Structure Enhanced T5), a model with a novel pretraining stage positioned between the initial T5 pretraining and the fine-tuning for the Text-to-SPARQL task. In this intermediary stage, we introduce a new objective called Triplet Structure Correction (TSC) to train the model on a SPARQL corpus derived from Wikidata. This objective aims to deepen the model’s understanding of the order of triplets. After this specialized pretraining, the model undergoes fine-tuning for SPARQL query generation, augmenting its query-generation capabilities. We also propose a method named “semantic transformation” to fortify the model’s grasp of SPARQL syntax and semantics without compromising the pre-trained weights of T5. Experimental results demonstrate that our proposed TSET outperforms existing methods on three well-established KBQA datasets: LC-QuAD 2.0, QALD-9 plus, and QALD-10, establishing a new state-of-the-art performance (95.0% F1 and 93.1% QM on LC-QuAD 2.0, 75.85% F1 and 61.76% QM on QALD-9 plus, 51.37% F1 and 40.05% QM on QALD-10).

show abstract

Section: Methodsmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 94%

See 1 more Smart Citation

Enhancing SPARQL Query Generation for Knowledge Base Question Answering Systems by Learning to Correct Triplets

Qi,

Su,

Guo

et al. 2024

Applied Sciences

View full text Add to dashboard Cite

show abstract

“…The CWQ [8] benchmark provides questions in Hebrew, Kannada, Chinese, and English, with the non-English questions translated by machine translation with manual adjustments. The QALD-9-plus benchmark [22] introduced improvements and an extension of the multilingual translations in its previous version -QALD-9 [39] -by involving crowd-workers with native-level language skills for high-quality translations from English to their native languages as well as validation. In addition, the authors manually transformed gold standard queries from DBpedia to Wikidata.…”

Section: Multilinguality In Kgqamentioning

confidence: 99%

QALD-10 – The 10th challenge on question answering over linked data

Usbeck,

Yan,

Perevalov

et al. 2023

Self Cite

View full text Add to dashboard Cite

Knowledge Graph Question Answering (KGQA) has gained attention from both industry and academia over the past decade. Researchers proposed a substantial amount of benchmarking datasets with different properties, pushing the development in this field forward. Many of these benchmarks depend on Freebase, DBpedia, or Wikidata. However, KGQA benchmarks that depend on Freebase and DBpedia are gradually less studied and used, because Freebase is defunct and DBpedia lacks the structural validity of Wikidata. Therefore, research is gravitating toward Wikidata-based benchmarks. That is, new KGQA benchmarks are created on the basis of Wikidata and existing ones are migrated. We present a new, multilingual, complex KGQA benchmarking dataset as the 10th part of the Question Answering over Linked Data (QALD) benchmark series. This corpus formerly depended on DBpedia. Since QALD serves as a base for many machine-generated benchmarks, we increased the size and adjusted the benchmark to Wikidata and its ranking mechanism of properties. These measures foster novel KGQA developments by more demanding benchmarks. Creating a benchmark from scratch or migrating it from DBpedia to Wikidata is non-trivial due to the complexity of the Wikidata knowledge graph, mapping issues between different languages, and the ranking mechanism of properties using qualifiers. We present our creation strategy and the challenges we faced that will assist other researchers in their future work. Our case study, in the form of a conference challenge, is accompanied by an in-depth analysis of the created benchmark.

show abstract

“…Other related datasets include QALM (Kaffee et al, 2019), a dataset for multilingual question answering over a set of different popular knowledge graphs, intended to help determine the multilinguality of those knowledge graphs. Similarly, QALD-9 (Ngomo, 2018) and QALD-9-plus (Perevalov et al, 2022a) support the development of multilingual question answering systems, tied to DBpedia and Wikidata, respectively. The goal of both datasets is to expand QA systems to more languages rather than improving compositionality.…”

Section: Related Workmentioning

confidence: 99%

Compositional Generalization in Multilingual Semantic Parsing over Wikidata

Cui

Aralikatte

Lent

et al. 2022

Transactions of the Association for Computational Linguistics

View full text Add to dashboard Cite

Semantic parsing (SP) allows humans to leverage vast knowledge resources through natural interaction. However, parsers are mostly designed for and evaluated on English resources, such as CFQ (Keysers et al., 2020), the current standard benchmark based on English data generated from grammar rules and oriented towards Freebase, an outdated knowledge base. We propose a method for creating a multilingual, parallel dataset of question-query pairs, grounded in Wikidata. We introduce such a dataset, which we call Multilingual Compositional Wikidata Questions (MCWQ), and use it to analyze the compositional generalization of semantic parsers in Hebrew, Kannada, Chinese, and English. While within- language generalization is comparable across languages, experiments on zero-shot cross- lingual transfer demonstrate that cross-lingual compositional generalization fails, even with state-of-the-art pretrained multilingual encoders. Furthermore, our methodology, dataset, and results will facilitate future research on SP in more realistic and diverse settings than has been possible with existing resources.

show abstract

QALD-9-plus: A Multilingual Dataset for Question Answering over DBpedia and Wikidata Translated by Native Speakers

Cited by 26 publications

References 0 publications

Enhancing SPARQL Query Generation for Knowledge Base Question Answering Systems by Learning to Correct Triplets

Enhancing SPARQL Query Generation for Knowledge Base Question Answering Systems by Learning to Correct Triplets

QALD-10 – The 10th challenge on question answering over linked data

Compositional Generalization in Multilingual Semantic Parsing over Wikidata

Contact Info

Product

Resources

About