Assessing relevance and trust of the deep web sources and results based on inter-source agreement

Abstract

Deep web search engines face the formidable challenge of retrieving high quality results from the vast collection of searchable databases. Deep web search is a two step process of selecting the high quality sources and ranking the results from the selected sources. Though there are existing methods for both the steps, they assess the relevance of the sources and the results using the query-result similarity. When applied to the deep web these methods have two deficiencies. First is that they are agnostic to the correctness (trustworthiness) of the results. Secondly, the query based relevance does not consider the importance of the results and sources. These two considerations are essential for the deep web and open collections in general. Since a number of deep web sources provide answers to any query, we conjuncture that the agreements between these answers are helpful in assessing the importance and the trustworthiness of the sources and the results. For assessing source quality, we compute the agreement between the sources as the agreement of the answers returned. While computing the agreement, we also measure and compensate for the possible collusion between the sources. This adjusted agreement is modeled as a graph with sources at the vertices. On this agreement graph, a quality score of a source that we call SourceRank, is calculated as the stationary visit probability of a random walk. For ranking results, we analyze the second order agreement between the results. Further extending SourceRank to multi-domain search, we propose a source ranking sensitive to the query domains. Multiple domain specific rankings of a source are computed, and these ranks are combined for the final ranking. We perform extensive evaluations on online and hundreds of Google Base sources spanning across domains. The proposed result and source rankings are implemented in the deep web search engine Factal. We demonstrate that the agreement analysis tracks source corruption. Further, our relevance evaluations show that our methods improve precision significantly over Google Base and the other baseline methods. The result ranking and the domain specific source ranking are evaluated separately.

On truth discovery in social sensing: A maximum likelihood estimation approach

“…The source dependency detection problem has been discussed and several solutions have been proposed [5], [10], [11]. Additionally, trust analysis has been done both on a homogeneous network [4], [27] and a heterogeneous network [23]. Our proposed EM scheme is the first piece of work that finds a maximum likelihood estimator to directly and optimally quantify the accuracy of conclusions obtained from credibility analysis in social sensing.…”

Knowledge Engineering Method Based on Consensual Knowledge and Trust Computation: The MUSCKA System

“…In the context of our experiment, before analysing in depth the sources, the experts believed that labels extracted from Agrovoc were going to be more relevant than labels from NCBI. Approaches such as [5] could be used to evaluate beforehand the strengths of the considered sources. In this paper we consider all candidates independently from one another.…”

“…It is available on github at https://github.com/Murloc6/Muskca. After analysing the results of the OAEI challenge 3 and especially the ones of the instance matching task 4 [10] dealing with tools mapping all kinds of ontological objects, we chose to use LogMap [11] of which the source code is available online 5 .…”

“…[9] highlights the fact that measures are needed in order to assign trust to specific pieces of a source, considering that when evaluating trust the information contained in the source can not be considered as a whole. As many propositions have been made in order to measure the trust to give to the sources themselves [5], this paper focuses on how trust can be evaluated for elements of a KB candidate for reuse in a specific context. …”

Approximate Truth Discovery via Problem Scale Reduction

“…Other related approaches include crowdsourcing [9] and web search [2,17]. Crowdsourcing systems assess worker quality in accomplishing their crowdsourcing tasks while web search methods recommend trustworthy source links to answer user's queries.…”