Evaluating question answering over linked data

Abstract

The availability of large amounts of open, distributed and structured semantic data on the web has no precedent in the history of computer science. In recent years, there have been important advances in semantic search and question answering over RDF data. In particular, natural language interfaces to online semantic data have the advantage that they can exploit the expressive power of Semantic Web data models and query languages, while at the same time hiding their complexity from the user. However, despite the increasing interest in this area, there are no evaluations so far that systematically evaluate this kind of systems, in contrast to traditional question answering and search interfaces to document spaces. To address this gap, we have set up a series of evaluation challenges for question answering over linked data. The main goal of the challenge was to get insight into the strengths, capabilities and current shortcomings of question answering systems as interfaces to query linked data sources, as well as benchmarking how these interaction paradigms can deal with the fact that the amount of RDF data available on the web is very large and heterogeneous with respect to the vocabularies and schemas used. Here we report on the results from the first and second of such evaluation campaigns. We also discuss how the second evaluation addressed some of the issues and limitations which arose from the first one, as well as the open issues to be addressed in future competitions.

Experiencing OptiqueVQS: a multi-paradigm and ontology-based visual query system for end users

“…Natural query interfaces (e.g., [48,91]) go beyond the keyword search; this is due to the fact that they interpret a query as whole and take linguistic considerations into account. However, they do not achieve much in terms of accuracy and completeness due to the ambiguity of natural language; some approaches try to alleviate this issue, such as by acquiring feedback and clarification through user dialogues (e.g., [25,58]), yet this increases the user cognitive load. Finally, visual query languages and systems (cf.…”

Graph-Based RDF Data Management

“…Usually, there are two stages in RDF Q/A systems: question understanding and query evaluation. Existing systems in the first stage translate a natural language question N into SPARQL queries [11,18,32], which are evaluated in the second stage. The focus of the existing solutions is on query understanding.…”

NL-Graphs: A Hybrid Approach toward Interactively Querying Semantic Data

“…These problems would, for instance, result from the difficulty of mapping user query terms to ontological ones or understanding complex questions such as those containing superlatives or advanced constraints. Therefore, five queries were selected from the DBpedia training and test data provided by the 2nd Open Challenge on Question Answering over Linked Data [13]. These queries are listed below:…”