Stochastic Judgments in the AHP: The Measurement of Rank Reversal Probabilities

Abstract

This paper presents a methodology for analyzing Analytic Hierarchy Process (AHP) rankings if the pairwise preference judgments are uncertain (stochastic). If the relative preference statements are represented by judgment intervals, rather than single values, then the rankings resulting from a traditional (deterministic) AHP analysis based on single judgment values may be reversed, and therefore incorrect. In the presence of stochastic judgments, the traditional AHP rankings may be stable or unstable, depending on the nature of the uncertainty. We develop multivariate statistical techniques to obtain both point estimates and confidence intervals of the rank reversal probabilities, and show how simulation experiments can be used as an effective and accurate tool for analyzing the stability of the preference rankings under uncertainty. If the rank reversal probability is low, then the rankings are stable and the decision maker can be confident that the AHP ranking is correct. However, if the likelihood of rank reversal is high, then the decision maker should interpret the AHP rankings cautiously, as there is a subtantial probability that these rankings are incorrect. High rank reversal probabilities indicate a need for exploring alternative problem formulations and methods of analysis. The information about the extent to which the ranking of the alternatives is sensitive to the stochastic nature of the pairwise judgments should be valuable information into the decision‐making process, much like variability and confidence intervals are crucial tools for statistical inference. We provide simulation experiments and numerical examples to evaluate our method. Our analysis of rank reversal due to stochastic judgments is not related to previous research on rank reversal that focuses on mathematical properties inherent to the AHP methodology, for instance, the occurrence of rank reversal if a new alternative is added or an existing one is deleted.

Using imprecise estimates for weights

“…Similarly probabilistic models for practical decision support are harder to find, although using the cumulative probability distribution (risk function) for each score and the identification of stochastic dominance has been proposed (Moskowitz, Tang and Lam 2000) as have the modelling of a probability distribution of the rank of each alternative (Bañuelas and Antony 2007;Jessop 2002;Butler, Jia and Dyer 1997) and the probability of rank reversal (Stam and Duarte Silva 1997;Saaty and Vargas 1987).…”

“…The most frequently cited method is the eigenvector model of Saaty (1977). A number of studies have used simulation to investigate the effects of uncertainty on the ranking of alternatives found in this way (Bañuelas and Antony 2007;Lipovetsky and Tishler 1999;Levary and Wan 1998;Stam and Duarte Silva 1997;Hauser and Tadikamalla 1996;Saaty and Vargas 1987;Vargas 1982). …”

Using intervals for global sensitivity and worst-case analyses in multiattribute value trees

“…Arbel and Vargas 1993, Butler et al 1997, Stam and Silva 1997. The main advantage of this method is that one gets a lot of information, such as mean values, variances and fractiles, about the characteristics of the decision model subject to uncertainties.…”

Stochastic pairwise comparative judgements and direct ratings of alternatives in the REMBRANDT system

“…Saaty and Vargas' method, whilst tractable, appears cumbersome and impractical: there is a requirement to perform a full-scale simulation each time uncertainty arises in a discrete multicriteria decision problem. Furthermore, Stam and Silva (1994) criticize Saaty and Vargas (1987) on several counts. For instance, (i) the method used to construct their impact score intervals is questioned (it is dependent on the level of confidence used) and (ii) the impact score interval for each component of the right principal eigenvector is computed independently of that for each other component, ignoring the possibility of correlation between components, which Stam and Silva show may not be insignificant.…”