Conference v2.0: An Uncertain Version of the OAEI Conference Benchmark

Cheatham, Michelle; Hitzler, Pascal

doi:10.1007/978-3-319-11915-1_3

Cited by 30 publications

(33 citation statements)

References 10 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The intent of this track is to provide expressive and real-world matching problems over expressive ontologies covering the same domain [2]. This data set is composed of 16 ontologies covering the domain of conference organization and a subset of 21 reference alignments involving 7 ontologies.…”

Section: Oaei Conference Data Setmentioning

confidence: 99%

An Extensible Linear Approach for Holistic Ontology Matching

Megdiche

Teste

Santos

2016

Lecture Notes in Computer Science

View full text Add to dashboard Cite

OATAO is an open access repository that collects the work of Toulouse researchers and makes it freely available over the web where possible. This is an author-deposited version published in : http://oatao.univ-toulouse.fr/ Eprints ID : 17170The contribution was presented at ISWC 2016 :http://iswc2016.semanticweb.org/ Abstract. Resolving the semantic heterogeneity in the semantic web requires finding correspondences between ontologies describing resources. In particular, with the explosive growth of data sets in the Linked Open Data, linking multiple vocabularies and ontologies simultaneously, known as holistic matching problem, becomes necessary. Currently, most state-of-the-art matching approaches are limited to pairwise matching. In this paper, we propose a holistic ontology matching approach that is modeled through a linear program extending the maximumweighted graph matching problem with linear constraints (cardinality, structural, and coherence constraints). Our approach guarantees the optimal solution with mostly coherent alignments. To evaluate our proposal, we discuss the results of experiments performed on the Conference track of the OAEI 2015, under both holistic and pairwise matching settings.

show abstract

Section: Oaei Conference Data Setmentioning

confidence: 99%

An Extensible Linear Approach for Holistic Ontology Matching

Megdiche

Teste

Santos

2016

Lecture Notes in Computer Science

View full text Add to dashboard Cite

show abstract

“…The final version of the game solves ontology alignment through two atomic tasks: (1) choosing a related concept -given a DBpedia concept players choose and agree upon a related PROTON concept; (2) specifying the type of relation between two concepts in terms of equivalence or subsumption. More recently, Cheatham and Hitzler [5] made use of MTurk to generate a mapping for the Conference track of the Ontology Alignment Evaluation Initiative (OAEI) thus pioneering the use of crowdsourcing for generating benchmark data in the Semantic Web research area. They conclude that crowdsourcing offers a scalable, cost-effective method for generating benchmarks that highly agree with expert opinion.…”

Section: Use Of Crowdsourcing For Knowledge Acquisitionmentioning

confidence: 99%

“…Presented with a pair of terms (typically representing ontology concepts) and a relation between these terms, crowd-workers judge whether the suggested relation holds. Frequently verified relations include generic ontology relations such as equivalence [5,25] and subsumption [19,25], which are relevant both in ontology evaluation [19] and ontology alignment scenarios [25]. T3.…”

Section: Typical Crowdsourcing Tasks In Ontology Engineeringmentioning

confidence: 99%

See 1 more Smart Citation

Crowd-based ontology engineering with the uComp Protégé plugin

Wohlgenannt

Sabou

Hanika

2016

View full text Add to dashboard Cite

Abstract. Crowdsourcing techniques provide effective means for solving a variety of ontology engineering problems. Yet, they are mainly used as external support to ontology engineering, without being closely integrated into the work of ontology engineers. In this paper we investigate how to closely integrate crowdsourcing into ontology engineering practices. Firstly, we show that a set of basic crowdsourcing tasks are used recurrently to solve a range of ontology engineering problems. Secondly, we present the uComp Protégé plugin that facilitates the integration of such typical crowdsourcing tasks into ontology engineering from within the Protégé ontology editor. An evaluation of the plugin in a typical ontology engineering scenario where ontologies are built from automatically learned semantic structures, shows that its use reduces the working times for the ontology engineers 11 times, lowers the overall task costs by 40% to 83% depending on the crowdsourcing settings used and leads to data quality comparable with that of tasks performed by ontology engineers. Evaluations on a large anatomy ontology confirm that crowdsourcing is a scalable and effective method: good quality results (accuracy of 89% and 99%) are obtained while achieving cost reductions of 75% from the ontology engineer costs and providing comparable overall task duration.

show abstract

“…In assessing the OAEI benchmark, [9] found that disagreement between annotators (both crowd and expert) is an indicator for inherent uncertainty in the domain knowledge, and that current benchmarks in ontology alignment and evaluation are not designed to model this uncertainty. [22] found similar results for the task of crowdsourced POS tagging -most inter-annotator disagreement was indicative of debatable cases in linguistic theory, rather than faulty annotation.…”

Section: State Of the Artmentioning

confidence: 99%

Crowdsourcing Disagreement for Collecting Semantic Annotation

Dumitrache¹

2015

Lecture Notes in Computer Science

View full text Add to dashboard Cite

Abstract. This paper proposes an approach to gathering semantic annotation, which rejects the notion that human interpretation can have a single ground truth, and is instead based on the observation that disagreement between annotators can signal ambiguity in the input text, as well as how the annotation task has been designed. The purpose of this research is to investigate whether disagreement-aware crowdsourcing is a scalable approach to gather semantic annotation across various tasks and domains. We propose a methodology for answering this question that involves, for each task and domain: defining the crowdsourcing setup, experimental data collection, and evaluating both the setup and the results. We present initial results for the task of medical relation extraction, and propose an evaluation plan for crowdsourcing semantic annotation for several tasks and domains.

show abstract

Conference v2.0: An Uncertain Version of the OAEI Conference Benchmark

Cited by 30 publications

References 10 publications

An Extensible Linear Approach for Holistic Ontology Matching

An Extensible Linear Approach for Holistic Ontology Matching

Crowd-based ontology engineering with the uComp Protégé plugin

Crowdsourcing Disagreement for Collecting Semantic Annotation

Contact Info

Product

Resources

About