Dynamically Updating Relevance Judgements in Probabilistic Information Systems Via Users' Feedback

Abstract

A decision maker's performance relies on the availability of relevant information. In many environments, the relation between the decision maker's informational needs and the information base is complex and uncertain. A fundamental concept of information systems, such as decision support and document retrieval, is the probability that the retrieved information is useful to the decision maker's query. This paper presents a sequential, Bayesian, probabilistic indexing model that explicitly combines expert opinion with data about the system's performance. The expert opinion is encoded into probability statements. These statements are modified by the users' feedback about the relevance of the retrieved information to their queries. The predictive probability that a datum in the information base is applicable to the current query is a logistic function of the expert opinion and the feedback. This feedback enters the computation through a measure of association between the current query-datum pair with previous, relevant query-datum pairs. When this measure is based on the proportional matching of multiple attributes, the predictive probabilities have a recursive formula that makes the model computationally feasible for large information bases.decision theory, Bayesian inference, decision support systems, expert systems, document retrieval, probabilistic indexing

When is the probability ranking principle suboptimal?

“…In fact, in situations involving many uncertain variables, each of which must be combined in making the calculation, the certainty we place on a given predictive probability may be negligible. Tague (1973) and Lenk and Floyd (1988) described the uncertainty in predictive probabilities of relevance by using probability distributions. The intent of these systems is to adequately model the uncertainty inherent in IR systems and to compensate for inadequate system design by incorporating the inquirer's judgements through Bayes theorem.…”

“…Instead, the inquirer's query and the documents' descriptions are sufficient information for the system to produce predictive probabilities, and additional information, such as the inquirer's relevance judgements about other documents, is superfluous. Other authors, such as Bookstein (1983) and Lenk and Floyd (1988), have considered models with dependent assessments.…”

“…This article critically examines possible difficulties with the standard policy, and the analysis applies to a variety of probabilistic learning and updating procedures, such as those described by Tague (1973), Bookstein (1983), Lenk and Floyd (1988), and Gordon (1991). The intent of this article is not to discredit probabilistic IR systems.…”

Computerized Decision Aids for Flexible Benefits Decisions: The Effects of an Expert System and Decision Support System on Employee Intentions and Satisfaction With Benefits

“…Not surprisingly, outside of the professional press, most of the ES research is reported in the management information system and computer science literatures. These studies focus on analyses of ES feasibility (Goul & Tong, 1987), case studies of applications (Sviokla, 1990), issues related to ease of use (Lamberti & Wallace, 1990), and evaluations of alternative methods of knowledge acquisition and representation (Hoffman, Shadbolt, Burton, & Klein, 1995;Lenk & Floyd, 1988;Liang, 1992;Tou, 1985;Wright & Ayton, 1987). Research into the effects of ESs on users reports that ESs generally improve individual performance and learning (Fedorowicz, Oz, & Berger, 1992;Lamberti & Wallace, 1990;Moffitt, 1994;Sviokla, 1990).…”

Modeling users' preferences in systems for information access

“…To improve the system's capability of correctly estimating the user preferences, a learning component called the relevance feedback mechanism often is incorporated within the basic structure of an IRS. 11,27,28 This component concentrates on allowing a retroaction controlled by the user in order to improve the retrieval results on the basis of further indications, which allow the system to learn the real user preferences. Figure 3 shows the scheme of an IRS with a relevance feedback mechanism.…”