A novel, unbiased approach to evaluating subsequent search misses in dual target visual search

Abstract

Research in radiology and visual cognition suggest that finding one target during visual search may result in increased misses for a second target, an effect known as subsequent search misses (SSM). Here, we demonstrate that the common method of calculating second-target detection performance is biased and could produce spurious SSM effects. We describe the source of that bias and document factors that influence its magnitude. We use a modification of signal-detection theory to develop a novel, unbiased method of calculating the expected value for dual-target performance under the null hypothesis. We then apply our novel method to two of our data sets that showed modest SSM effects when calculated in the traditional manner. Our correction reduced the effect size to the point that there was no longer a significant SSM effect. We then applied our method to a published data set that had a larger effect size when calculated using the traditional calculation as well as when using an alternative calculation that was recently proposed to account for biases in the traditional method. We find that both the traditional method and the recently proposed alternative substantially overestimate the magnitude of the SSM effect in these data, but a significant SSM effect persisted even with our calculation. We recommend that future SSM studies use our method to ensure accurate effect-size estimates, and suggest that the method be applied to reanalyze published results, particularly those with small effect sizes, to rule out the possibility that they were spurious.

Great expectations: minor differences in initial instructions have a major impact on visual search in the absence of feedback

“…In the current study, there was no significant difference in dual-target accuracy (defined as the percentage of dual-target trials where both targets where found) between groups (low-expectation: mean = 56.16%, SD = 14.23%, high-expectation: mean = 53.09%, SD = 18.76%; t (82) = − 0.84; p = 0.41) and a Bayes factor analysis found substantial evidence in favor of the null hypothesis ( B 01 = 4.31). However, measuring the SSM effect involves specifically comparing accuracy for a second-target to a single-target accuracy baseline (a number of different metrics have recently been proposed for measuring the SSM effect, see Adamo et al, 2019 ; Becker et al, 2020 ). Therefore, despite the lack of a significant difference in dual-target accuracy, there was a significant difference in the measures of the SSM effect between groups using all of the metrics from both Adamo et al, 2019 and Becker et al 2020 .…”

“…However, measuring the SSM effect involves specifically comparing accuracy for a second-target to a single-target accuracy baseline (a number of different metrics have recently been proposed for measuring the SSM effect, see Adamo et al, 2019 ; Becker et al, 2020 ). Therefore, despite the lack of a significant difference in dual-target accuracy, there was a significant difference in the measures of the SSM effect between groups using all of the metrics from both Adamo et al, 2019 and Becker et al 2020 . Critically, these differences were driven by the difference in the single-target baseline, so reffering to them as true differences in the SSM effect is complicated (see discussion).…”

“…This change in single-target performance without a change in dual-target performance has implications for calculating and reasoning about SSM error rates. Given that measures of the SSM effect (or multiple target search difficulty in general) compare dual-target performance to a single-target baseline (Adamo et al, 2019 ; Becker et al, 2020 ), the SSM metrics showed a larger effect for the high-expectation group than the low-expectation group despite no difference in the dual-target performance. Whether or not this should be truly classified as an increase in the SSM effect is a potentially interesting point for debate for those interested in properly quantifying SSM errors.…”

From “satisfaction of search” to “subsequent search misses”: a review of multiple-target search errors across radiology and cognitive science

“…More recently Becker et al ( 2020 ) argued the above approach is limited by its restrictive experimental design (i.e., that using matched displays can limit the number of dual-target trials within an experiment as there has to be two matched single-target trials). Furthermore, they hypothesized that stimulus matching is an imperfect correction for biases caused by differences in second-target detection difficulty and argued that biases could exist beyond those induced by stimulus display properties.…”

“…Becker et al ( 2020 ) demonstrated that by using this equation on the data reported in Adamo et al ( 2019 ) (which is based on the same displays and targets used in many prior SSM studies; e.g., Adamo et al, 2013 , 2017 , 2018 ; Cain & Mitroff, 2013 ), this approach results in a smaller, but still significant, SSM effect compared to both the non-matched and matched search display design/data tested in Adamo et al ( 2019 ) (see Becker et al, 2020 and Table 2 below for examples of how the SSM effect sizes change with different calculation methods).…”