Previous work shows that listeners use the phonetic signal to predict upcoming information, including adaptation to accent-specific phonological structures. This study assesses whether this response is optimal by studying the time course and specificity of adaptation as native French listeners are presented with English-accented French samples in a cross-modal two-alternative forced choice task. There were four types of trials, depending on whether the spoken target word contained a sound replaced in English-accented speech with a different phoneme; and whether this pronunciation rendered the choice temporarily ambiguous or not. If listeners follow an optimal strategy, one predicts an interaction between both terms, with slower responses for words with replaced phonemes only when the replacement rendered the target temporarily ambiguous. Instead, we found main effects of both factors. Thus, it appears that adult listeners did not adopt the strategy that would have been informationally optimal. This result could be due to prior experience with the accent employed, and specific difficulties with replaced phonemes, despite knowledge of the accent. We discuss results in the broader context of accent perception research, bearing on the (speaker- and contrast-)specificity of listener adaptation as well as the variability in results potentially due to the task, accent, and materials employed.