How to do Better N400 Studies: Reproducibility, Consistency and Adherence to Research Standards in the Existing Literature

Abstract

Given the complexity of ERP recording and processing pipeline, the resulting variability of methodological options, and the potential for these decisions to influence study outcomes, it is important to understand how ERP studies are conducted in practice and to what extent researchers are transparent about their data collection and analysis procedures. The review gives an overview of methodology reporting in a sample of 132 ERP papers, published between January 1980 – June 2018 in journals included in two large databases: Web of Science and PubMed. Because ERP methodology partly depends on the study design, we focused on a well-established component (the N400) in the most commonly assessed population (healthy neurotypical adults), in one of its most common modalities (visual images). The review provides insights into 73 properties of study design, data pre-processing, measurement, statistics, visualization of results, and references to supplemental information across studies within the same subfield. For each of the examined methodological decisions, the degree of consistency, clarity of reporting and deviations from the guidelines for best practice were examined. Overall, the results show that each study had a unique approach to ERP data recording, processing and analysis, and that at least some details were missing from all papers. In the review, we highlight the most common reporting omissions and deviations from established recommendations, as well as areas in which there was the least consistency. Additionally, we provide guidance for a priori selection of the N400 measurement window and electrode locations based on the results of previous studies.

Multiple functions of the angular gyrus at high temporal resolution

“…In the same way, differences in data acquisition and analysis procedures (Darvas et al, 2004;Puce and Hamalainen, 2017) are also present. A recent review identified more than 70 properties about design, data preprocessing, measurement, and statistics that can explain the lack of consistency across previous studies (Soskic et al, 2021). Such differences will inevitably yield differences in the exact timing and amplitude of ERP components and in the exact number and locations of brain sources.…”

Open and reproducible neuroimaging: from study inception to publication

“…A recent comprehensive investigation, the Neuroimaging Analysis Replication and Prediction Study (NARPS; Botvinik-Nezer et al 2020) found that when 70 different teams were asked to analyze the same fMRI data to test the same hypotheses, each team chose a distinct pipeline and results were highly variable. Other studies suggest similar problems in EEG (Šoškić et al 2021;Clayson et al 2021), PET (Nørgaard et al 2020) and diffusion MRI (Schilling et al 2021).…”

“…maximize predictive models, recent efforts in fMRI (task fMRI: (Botvinik-Nezer et al 2020; Carp 2012a); preprocessing of resting-state fMRI: (Li et al 2021)) and PET (specifically for preprocessing: Nørgaard et al 2020) focused more on the variability of outcomes in general when analysis pipelines were varied. In addition, recent studies showed high variability in diffusion-based tractography dissection (Schilling et al 2021) and event-related potentials in EEG preprocessing (Šoškić et al 2021;Clayson et al 2021). Another large-scale attempt to estimate the analytical variability for EEG, EEGManyPipelines (see Table S1), is currently ongoing.…”

Parafoveal and foveal N400 effects in natural reading: A timeline of semantic processing from fixation‐related potentials

“…However, subsequent analyses of N400 effects were performed on ERPs converted to an average‐mastoid reference (mean of M1 and M2). Because the mastoid reference electrodes are placed on the opposite (positive) side of the N400 scalp distribution, they often maximize N400 amplitudes and are therefore commonly used in N400 studies (Šoškić et al., 2022). In contrast, the earlier occipito‐temporal preview validity effect following the N1 component is best captured with an average reference (see Li et al., 2015, their figure 5).…”