Reflections on five years of evaluating semantic search systems

Uren, Victoria; Sabou, Marta; Motta, Enrico; Fernández, Miriam; López, Vanessa; Lei, Yuangui

doi:10.1504/ijmso.2010.033280

Cited by 14 publications

(10 citation statements)

References 21 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…In contrast with the Information Retrieval (IR) community, where evaluation using standardized techniques, such as those used for the annual TREC competitions, has been common for decades, the SW community is still a long way from defining standard evaluation benchmarks to evaluate the quality of semantic technologies [20]. Important efforts have been made in the last few years towards the establishment of common datasets, methodologies and metrics to evaluate semantic technologies, e.g., the SEALS project [26].…”

Section: Evaluation and Resultsmentioning

confidence: 99%

See 1 more Smart Citation

PowerAqua: Supporting users in querying and exploring the Semantic Web

et al. 2012

Self Cite

View full text Add to dashboard Cite

With the continued growth of online semantic information, the processes of searching and managing this massive scale and heterogeneous content have become increasingly challenging. In this work, we present PowerAqua, an ontologybased Question Answering system that is able to answer queries by locating and integrating information, which can be distributed across heterogeneous semantic resources. We provide a complete overview of the system including: the research challenges that it addresses, its architecture, the evaluations that have been conducted to test it, and an in-depth discussion showing how PowerAqua effectively supports users in querying and exploring Semantic Web content.

show abstract

Section: Evaluation and Resultsmentioning

confidence: 99%

“…These tables are known as Entity Mapping Tables (ETMs). In our example, the EMT for the query term "actors" contains, among several others, exact candidate matches in DBpedia, the movie database 20…”

Section: The Element Mapping Component: Powermapmentioning

confidence: 99%

PowerAqua: Supporting users in querying and exploring the Semantic Web

et al. 2012

Self Cite

View full text Add to dashboard Cite

show abstract

“…In this section, we present some example queries 13 to justify our claim that we can obtain answers to queries directly from the DBpedia semantically rich information -even in its current form. The solutions in Section 5 had allowed us to improve PowerAqua mapping and fusion algorithms to exhibit better performance, measured in terms of speed or seconds to answer a query, by shifting the focus on precision while minimizing the loss in recall.…”

Section: Initial Experiments and Discussionmentioning

confidence: 99%

“…A well-known limitation of the SW is its sparseness, as stated in [13], without a well populated SW, developing semantic search systems is only an intellectual exercise. Only a reduced number of topics were covered entirely or partially by an existing ontology (domain sparseness), and added to this, sparseness at the level of instances and relation (model complexity) was also found [4,10].…”

Section: Scaling To Highly Populated and Dense Ontologiesmentioning

confidence: 99%

Scaling Up Question-Answering to Linked Data

López

Nikolov

Sabou

et al. 2010

Knowledge Engineering and Management by the Masses

Self Cite

View full text Add to dashboard Cite

Abstract. Linked Data semantic sources, in particular DBpedia, can be used to answer many user queries. PowerAqua is an open multi-ontology Question Answering (QA) system for the Semantic Web (SW). However, the emergence of Linked Data, characterized by its openness, heterogeneity and scale, introduces a new dimension to the Semantic Web scenario, in which exploiting the relevant information to extract answers for Natural Language (NL) user queries is a major challenge. In this paper we discuss the issues and lessons learned from our experience of integrating PowerAqua as a front-end for DBpedia and a subset of Linked Data sources. As such, we go one step beyond the state of the art on end-users interfaces for Linked Data by introducing mapping and fusion techniques needed to translate a user query by means of multiple sources. Our first informal experiments probe whether, in fact, it is feasible to obtain answers to user queries by composing information across semantic sources and Linked Data, even in its current form, where the strength of Linked Data is more a by-product of its size than its quality. We believe our experiences can be extrapolated to a variety of end-user applications that wish to scale, open up, exploit and re-use what possibly is the greatest wealth of data about everything in the history of Artificial Intelligence.

show abstract

“…However, for natural language based question answering tools over linked data there are no systematic and standard evaluation benchmarks in place yet. Therefore, evaluations of such systems are typically small-scale and idiosyncratic in the sense that they are specific for certain settings or applications [38].…”

Section: Introductionmentioning

confidence: 99%

Evaluating Question Answering Over Linked Data

et al. 2013

Self Cite

View full text Add to dashboard Cite

The availability of large amounts of open, distributed and structured semantic data on the web has no precedent in the history of computer science. In recent years, there have been important advances in semantic search and question answering over RDF data. In particular, natural language interfaces to online semantic data have the advantage that they can exploit the expressive power of Semantic Web data models and query languages, while at the same time hiding their complexity from the user. However, despite the increasing interest in this area, there are no evaluations so far that systematically evaluate this kind of systems, in contrast to traditional question answering and search interfaces to document spaces. To address this gap, we have set up a series of evaluation challenges for question answering over linked data. The main goal of the challenge was to get insight into the strengths, capabilities and current shortcomings of question answering systems as interfaces to query linked data sources, as well as benchmarking how these interaction paradigms can deal with the fact that the amount of RDF data available on the web is very large and heterogeneous with respect to the vocabularies and schemas used. Here we report on the results from the first and second of such evaluation campaigns. We also discuss how the second evaluation addressed some of the issues and limitations which arose from the first one, as well as the open issues to be addressed in future competitions.

show abstract

Reflections on five years of evaluating semantic search systems

Cited by 14 publications

References 21 publications

PowerAqua: Supporting users in querying and exploring the Semantic Web

PowerAqua: Supporting users in querying and exploring the Semantic Web

Scaling Up Question-Answering to Linked Data

Evaluating Question Answering Over Linked Data

Contact Info

Product

Resources

About