Pay-As-You-Go Multi-user Feedback Model for Ontology Matching

Cruz, Isabel F.; Loprete, Francesco; Palmonari, Matteo; Stroe, Cosmin; Taheri, Aynaz

doi:10.1007/978-3-319-13704-9_7

Cited by 15 publications

(14 citation statements)

References 12 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Finally, this paper extends our previous conference paper [3] as follows. First, we use new measures to evaluate the results, including AUGC (Area Under the Gain Curve) and AURC (Area Under the Robustness Curve).…”

Section: Multi-user Feedbacksupporting

confidence: 67%

Quality-based model for effective and robust multi-user pay-as-you-go ontology matching1

Cruz

Palmonari

Loprete

et al. 2016

Self Cite

View full text Add to dashboard Cite

Using a pay-as-you-go strategy, we allow for a community of users to validate or invalidate mappings obtained by an automatic ontology matching system using consensus for each mapping. The ultimate objectives are effectiveness-improving the quality of the obtained alignment (set of mappings) measured in terms of Fmeasure as a function of the number of user interactions-and robustness-making the system as much as possible impervious to user validation errors. Our strategy consists of two major steps: candidate mapping selection, which ranks mappings based on their perceived quality so that users are presented first with those mappings with lowest quality, and feedback propagation, which seeks to validate or invalidate those mappings that are perceived to be similar to the mappings already presented to the users for validation. The purpose of these two strategies is twofold: achieve greater improvements earlier and minimize overall user interaction. There are three important features of our approach: the use of a dynamic ranking mechanism to adapt to the new conditions after each user interaction, the presentation of each mapping for validation more than once-revalidation-because of possible user errors, and the immediate propagation of the user input on a mapping without first achieving consensus for that mapping. We study extensively the effectiveness and robustness of our approach as several of these parameters change, namely the error and revalidation rates, as a function of the number of iterations, to provide conclusive guidelines for the design and implementation of multiuser feedback ontology matching systems.

show abstract

Section: Multi-user Feedbacksupporting

confidence: 67%

Quality-based model for effective and robust multi-user pay-as-you-go ontology matching1

Cruz

Palmonari

Loprete

et al. 2016

Self Cite

View full text Add to dashboard Cite

show abstract

“…Conventional crowdsourcing solutions for execution of collective tasks are based on choice questions where the possible task answers are predefined and the worker executes the task by choosing the preferred option among those available (Sarasua et al ., 2012; Cruz et al ., 2014; Cheatham & Hitzler, 2014). In this context, a widely adopted consensus evaluation mechanism is based on majority voting, meaning that the most selected option becomes the final task answer (Castano et al ., 2015).…”

Section: Crowdsourcing Techniquesmentioning

confidence: 99%

“…A similar solution is proposed in Cruz et al . (2014) where mapping validation is enforced through a pay-as-you-go approach characterized by the use of crowd feedback. A propagation mechanism is also presented in Cruz et al .…”

Section: Related Workmentioning

confidence: 99%

See 1 more Smart Citation

Crowd-assessing quality in uncertain data linking datasets

Faria¹,

Ferrara²,

Jiménez-Ruiz

et al. 2020

The Knowledge Engineering Review

View full text Add to dashboard Cite

Abstract The quality of a dataset used for evaluating data linking methods, techniques, and tools depends on the availability of a set of mappings, called reference alignment, that is known to be correct. In particular, it is crucial that mappings effectively represent relations between pairs of entities that are indeed similar due to the fact that they denote the same object. Since the reliability of mappings is decisive in order to perform a fair evaluation of automatic linking methods and tools, we call this property of mappings as mapping fairness. In this article, we propose a crowd-based approach, called Crowd Quality (CQ), for assessing the quality of data linking datasets by measuring the fairness of the mappings in the reference alignment. Moreover, we present a real experiment, where we evaluate two state-of-the-art data linking tools before and after the refinement of the reference alignment based on the CQ approach, in order to present the benefits deriving from the crowd assessment of mapping fairness.

show abstract

“…The post-matching reconciliation process has received considerable attention in the literature for schemas and ontologies. The systems in [7,8,9,71,72,73] rely on one user only, whereas the frameworks in [74,75,36,76,10,35,77,78] rely on multiple users. Although we focus on reconciliation with a single expert, our framework is extensible as the underlying probabilistic model is independent of the number of users.…”

Section: Related Workmentioning

confidence: 99%

Handling probabilistic integrity constraints in pay-as-you-go reconciliation of data models

Hùng

Weidlich

Tâm

et al. 2019

Information Systems

View full text Add to dashboard Cite

Data models capture the structure and characteristic properties of data entities, e.g., in terms of a database schema or an ontology. They are the backbone of diverse applications, reaching from information integration, through peer-to-peer systems and electronic commerce to social networking. Many of these applications involve models of diverse data sources. Effective utilisation and evolution of data models, therefore, calls for matching techniques that generate correspondences between their elements. Various such matching tools have been developed in the past. Yet, their results are often incomplete or erroneous, and thus need to be reconciled, i.e., validated by an expert. This paper analyses the reconciliation process in the presence of large collections of data models, where the network induced by generated correspondences shall meet consistency expectations in terms of integrity constraints. We specifically focus on how to handle data models that show some internal structure and potentially differ in terms of their assumed level of abstraction. We argue that such a setting calls for a probabilistic model of integrity constraints, for which satisfaction is preferred, but not required. In this work, we present a model for probabilistic constraints that enables reasoning on the correctness of individual correspondences within a network of data models, in order to guide an expert in the validation process. To support pay-as-you-go reconciliation, we also show how to construct a set of high-quality correspondences, even if an expert validates only a subset of all generated correspondences. We demonstrate the efficiency of our techniques for real-world datasets comprising database schemas and ontologies from various application domains.

show abstract

Pay-As-You-Go Multi-user Feedback Model for Ontology Matching

Cited by 15 publications

References 12 publications

Quality-based model for effective and robust multi-user pay-as-you-go ontology matching1

Quality-based model for effective and robust multi-user pay-as-you-go ontology matching1

Crowd-assessing quality in uncertain data linking datasets

Handling probabilistic integrity constraints in pay-as-you-go reconciliation of data models

Contact Info

Product

Resources

About