The PI IRAP : An alternative scoring algorithm for the IRAP using a probabilistic semiparametric effect size measure

Abstract

The Implicit Relational Assessment Procedure (IRAP) has been used to assess the probability of arbitrarily applicable relational responding or as an indirect measure of implicit attitudes. To date, IRAP effects have commonly been quantified using the DIRAP scoring algorithm, which was derived from Greenwald, Nosek and Banaji's (2003) D effect size measure. In the article, we highlight the difference between an effect size measure and a scoring algorithm, discuss the drawbacks associated with D, and propose an alternative: a probabilistic, semiparametric measure referred to as the Probabilistic Index (Thas, De Neve, Clement, & Ottoy, 2012). Using a relatively large IRAP dataset, we demonstrate how the PI is more robust to the influence of outliers and skew (which are typical of reaction time data). Finally, we conclude that PI models, in addition to producing point estimate scores, can also provide confidence intervals, significance tests, and afford the possibility to include covariates, all of which may aid single subject design studies.

Pets and Politics: Do Liberals and Conservatives Differ in Their Preferences for Cats Versus Dogs?

“…In our case, a positive D score represents a faster categorization for cats when paired with positive words over negative words, implying a more positive attitude towards cats compared to dogs. Although the D measure is the original scoring and the one employed most often, others have argued that there are superior ways to score the IAT that are more robust to outliers and skew (De Schryver et al, 2018). Thus, we repeated our analyses using the PI, which gives the probability that a trial's response time is faster than another trial, with PI scores ranging from 0 to 1.…”

“…And a score of 0.5 means there is no difference between the two. Note that the PI only gives information about which trial is likely to be responded to more quickly, but not the magnitude of this difference (De Schryver et al, 2018).…”

Cultural pressure and biased responding in free will attitudes

“…To test whether participants had positive/negative attitudes towards free will/determinism (Hypothesis 1, see table 1 ), we assessed implicit and explicit FWAs. In order to assess implicit FWAs, we used the IAT A score (also referred to as Ruscio's A or the probability of superiority [ 34 ]), which is of higher psychometric quality than the commonly reported IAT D score [ 31 , 35 ]. In either case, IAT scores reflect the degree to which a participant is quicker to respond when ‘free will’ shares a response mapping with positive words and ‘determinism’ with negative words relative to the opposite mapping.…”

The Implicit Relational Assessment Procedure demonstrates poor internal consistency and test-retest reliability: A meta-analysis

“…Use a more robust scoring method. Recent research has argued that the D score is overly sensitive to the outliers that are frequently observed in reaction time data (De Schryver et al, 2018), and has suggested a more robust scoring method as an alternative. This method has been referred to by several names, including the Probabilistic Index, the Probability of Superiority and Ruscio's A (Ruscio, 2008).…”