RuBQ: A Russian Dataset for Question Answering over Wikidata

Korablinov, Vladislav; Braslavski, Pavel

doi:10.1007/978-3-030-62466-8_7

Cited by 13 publications

(7 citation statements)

References 35 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The annotation was performed via Yandex.Toloka ( ), a Russian crowd-sourcing platform with a high share of Russian speaking workers. Yandex.Toloka is widely used in the studies on Russian-language content, such as for annotation of semantic change ( Rodina & Kutuzov, 2020 ), question answering ( Korablinov & Braslavski, 2020 ), and toxic comments ( Smetanin, 2020b ). A depiction of the Yandex.Toloka user interface can be found in Fig.…”

Section: Sentiment Datasetmentioning

confidence: 99%

RuSentiTweet: a sentiment analysis dataset of general domain tweets in Russian

Smetanin¹

2022

PeerJ Computer Science

View full text Add to dashboard Cite

The Russian language is still not as well-resourced as English, especially in the field of sentiment analysis of Twitter content. Though several sentiment analysis datasets of tweets in Russia exist, they all are either automatically annotated or manually annotated by one annotator. Thus, there is no inter-annotator agreement, or annotation may be focused on a specific domain. In this article, we present RuSentiTweet, a new sentiment analysis dataset of general domain tweets in Russian. RuSentiTweet is currently the largest in its class for Russian, with 13,392 tweets manually annotated with moderate inter-rater agreement into five classes: Positive, Neutral, Negative, Speech Act, and Skip. As a source of data, we used Twitter Stream Grab, a historical collection of tweets obtained from the general Twitter API stream, which provides a 1% sample of the public tweets. Additionally, we released a RuBERT-based sentiment classification model that achieved F1 = 0.6594 on the test subset.

show abstract

Section: Sentiment Datasetmentioning

confidence: 99%

RuSentiTweet: a sentiment analysis dataset of general domain tweets in Russian

Smetanin¹

2022

PeerJ Computer Science

View full text Add to dashboard Cite

show abstract

“…Other KGQA datasets are Free917 (Cai and Yates, 2013), WebQuestions (Berant et al, 2013), Com-plexQuestions (Bao et al, 2016), SimplesQuestions (Bordes et al, 2015), GraphQuestions (Su et al, 2016), WebQuestionsSP (Yih et al, 2016), 30MFactoidQA (Serban et al, 2016), ComplexWebQuestions (Talmor and Berant, 2018), PathQuestion (Zhou et al, 2018), MetaQA (Zhang et al, 2018), TempQuestions (Jia et al, 2018), TimeQuestions (Jia et al, 2021), Cron-Questions (Saxena et al, 2021), FreebaseQA (Jiang et al, 2019), Compositional Freebase Questions (CFQ) (Keysers et al, 2019), Compositional Wikidata Questions (CWQ) (Cui et al, 2021), RuBQ (Korablinov and Braslavski, 2020;Rybin et al, 2021), GrailQA (Gu et al, 2021), Event-QA (Souza Costa et al, 2020), Sim-pleDBpediaQA (Azmy et al, 2018), CLC-QuAD (Zou et al, 2021), KQA Pro (Shi et al, 2020), SimpleQues-tionsWikidata (Diefenbach et al, 2017), DBNQA (Yin et al, 2019), etc. These datasets do not fulfill our current criteria and thus are not part of the initial version of the KGQA leaderboard.…”

Section: Kgqa Datasetsmentioning

confidence: 99%

Knowledge Graph Question Answering Leaderboard: A Community Resource to Prevent a Replication Crisis

Perevalov¹,

Yan²,

Kovriguina³

et al. 2022

Preprint

View full text Add to dashboard Cite

Data-driven systems need to be evaluated to establish trust in the scientific approach and its applicability. In particular, this is true for Knowledge Graph (KG) Question Answering (QA), where complex data structures are made accessible via natural-language interfaces. Evaluating the capabilities of these systems has been a driver for the community for more than ten years while establishing different KGQA benchmark datasets. However, comparing different approaches is cumbersome. The lack of existing and curated leaderboards leads to a missing global view over the research field and could inject mistrust into the results. In particular, the latest and most-used datasets in the KGQA community, LC-QuAD and QALD, miss providing central and up-to-date points of trust. In this paper, we survey and analyze a wide range of evaluation results with significant coverage of 100 publications and 98 systems from the last decade. We provide a new central and open leaderboard for any KGQA benchmark dataset as a focal point for the community -https://kgqa.github.io/leaderboard/. Our analysis highlights existing problems during the evaluation of KGQA systems. Thus, we will point to possible improvements for future evaluations.

show abstract

“…A wide range of KGQA benchmarks and datasets as well as analyses thereof have been created to evaluate KGQA systems for simple and complex questions over different publicly available Knowledge Graphs (KGs). This includes datasets such as WebQuestions [4] and GrailQA [14] for Freebase, LC-QuAD [32] and QALD-9 [34] for DBpedia, RuBQ [21] and CronQuestions [25] for Wikidata. However, none of them are geared towards generalization in KGQA.…”

Section: Related Workmentioning

confidence: 99%

Knowledge Graph Question Answering Datasets and Their Generalizability: Are They Enough for Future Research?

Jiang¹,

Usbeck²

2022

Preprint

View full text Add to dashboard Cite

Existing approaches on Question Answering over Knowledge Graphs (KGQA) have weak generalizability. That is often due to the standard i.i.d. assumption on the underlying dataset. Recently, three levels of generalization for KGQA were defined, namely i.i.d., compositional, zero-shot. We analyze 25 well-known KGQA datasets for 5 different Knowledge Graphs (KGs). We show that according to this definition many existing and online available KGQA datasets are either not suited to train a generalizable KGQA system or that the datasets are based on discontinued and out-dated KGs. Generating new datasets is a costly process and, thus, is not an alternative to smaller research groups and companies. In this work, we propose a mitigation method for re-splitting available KGQA datasets to enable their applicability to evaluate generalization, without any cost and manual effort. We test our hypothesis on three KGQA datasets, i.e., LC-QuAD, LC-QuAD 2.0 and QALD-9). Experiments on re-splitted KGQA datasets demonstrate its effectiveness towards generalizability. The code and a unified way to access 18 available datasets is online at https://github.com/semantic-systems/KGQA-datasets as well as https://github.com/semantic-systems/KGQA-datasetsgeneralization.

show abstract

RuBQ: A Russian Dataset for Question Answering over Wikidata

Cited by 13 publications

References 35 publications

RuSentiTweet: a sentiment analysis dataset of general domain tweets in Russian

RuSentiTweet: a sentiment analysis dataset of general domain tweets in Russian

Knowledge Graph Question Answering Leaderboard: A Community Resource to Prevent a Replication Crisis

Knowledge Graph Question Answering Datasets and Their Generalizability: Are They Enough for Future Research?

Contact Info

Product

Resources

About