Outlier exclusion procedures for reaction time analysis: The cures are generally worse than the disease.

Abstract

A methodological problem in most reaction time (RT) tasks is that some measured RTs may be outliers, being either too fast or too slow to reflect the task-related processing of interest. Numerous ad hoc procedures have been used to identify these outliers for exclusion from further analyses, but the accuracies of these methods have not been systematically compared. The present study compared the performance of 58 different outlier exclusion procedures (OEPs) using four huge datasets of real RTs. The results suggest that these OEPs are likely to do more harm than good, because they incorrectly identify outliers, increase noise, introduce bias, and generally reduce statistical power. The results suggest that RT researchers should not automatically apply any of these OEPs to clean their RT data prior to the main analyses.

A novel demonstration of preparation in pop-out search.

“…Only trials with correct responses to the single-item and search display were included in RT analyses. We excluded RTs below 200 ms and above 3,500 ms (0.4% of data) as they were unlikely to represent task-relevant processes (Miller, 2023). The means of the remaining search RTs for each condition are shown in the left panel of Figure 2.…”

Check your outliers! An introduction to identifying statistical outliers in R with easystats

“…For example, in the case of reaction time analysis, Miller (2023) systematically compared 58 SOD procedures in simulations using large datasets of real reaction times. He concluded that regardless of the selected procedure, the exclusion of outliers (reaction times too slow or too fast) generally did more harm than good compared to retaining them.…”

“…Setting an upper limit on very long times (e.g., 3 to 5 seconds, depending on the experimental task) to remove potential sparse artifacts, can also improve model convergence and fitting. Miller (2023) also suggests that it is generally better to assess outliers within specific experimental conditions or groups (a condition-specific strategy), rather than across the entire dataset at once (a pooled strategy), particularly in the case of reaction times. Additionally, common procedures such as statistical transformations (e.g.…”

Sclodnick, Sun, & Milliken; A Novel Demonstration of Preparation in Pop-out Search

“…Only trials with correct responses to the single-item and search display were included in response time (RT) analyses. We excluded RTs below 200 ms and above 3500 ms (0.4% of data) as they were unlikely to represent task-relevant processes (Miller, 2023). Means of the remaining search RTs for each condition are shown in the left panel of Figure 2.…”