Raw data and analysis scripts of "Quality Over Quantity: Focusing on High-Conflict Trials to Improve the Reliability and Validity of Attentional Control Measures"

In conflict tasks, congruency effects are thought to reflect attentional control mechanisms needed to counteract response conflict elicited by incongruent stimuli. Although the congruency effect is a well-replicable experimental effect, recent studies have evidenced low correlations between congruency effects measured across different paradigms, leading to a heated debate over whether these low correlations indicate a lack of construct validity or are rather attributable to high measurement error, as indicated by the poor reliability typically displayed by congruency effects. In the present study, we investigated whether the poor reliability of the congruency effect is due to its poor theoretical specification. Specifically, we tested whether the psychometric properties of congruency effects can be improved by focusing exclusively on those trials in which response conflict is theoretically expected to be highest. We considered two factors modulating the degree of response conflict: previous trial congruency, with higher conflict following congruent trials, and the time elapsed since stimulus onset, with higher conflict in fast responses. Data from 195 participants completing a Simon and a spatial Stroop paradigm showed that generally poor split-half reliabilities for the full set of trials improved greatly when excluding post-congruent and slow trials. Importantly, between-task correlations also increased substantially when controlling for these factors, suggesting that, with increased reliability, these tasks capture a common attentional control ability. Our results suggest that individual differences in attentional control can be more reliably measured when focusing on the trials with the highest response conflict. Therefore, dismissing attentional control as an ability may be premature.