The Role of Exchangeability in Sequential Updating of Findings from Small Studies and The Challenges of Identifying Exchangeable Data sets

The Impact of Sample Size on Exchangeability in the Bayesian Synthesis Approach to Data Fusion

“…The observed relative bias for the variance of the intercept, the slope, and to a lesser degree the intercept-slope covariance in this first data design condition highlight the importance that the order of fusing the datasets can play in the estimation of parameters when implementing Bayesian Synthesis strategies. Interestingly, this finding is not in line with past research that has suggested that the order of data fusion does not meaningfully impact the final posterior distribution results (Marcoulides, 2017b;Miocevic et al, 2020). When the data sets being fused are of differing sizes (50 vs. 1,000), ending with the fusion and analysis of a large dataset can in fact produce a substantially biased final posterior distribution when the other sequentially analyzed datasets are much smaller.…”

“…To address the concern about the order of the datasets being analyzed, Marcoulides (2017b) examined the exchangeability assumption and found that the order of analysis did not meaningfully impact the final data fusion results. Similar conclusions regarding exchangeability were also recently suggested by Miocevic, Levy, and Savord (2020). One limitation with these conclusions is that they were based on analyzed datasets that were from similarly-sized large samples.…”

Bayesian mediation analysis in trauma research.

“…Importantly, in rjags, when specifying an informative gamma prior for the residual precision based on an observed posterior, it is recommended that half of the previous sample size ( N /2) be used for the scale parameter, and half the product between the previous sample size and corresponding variance parameter, ( N × σ 2 )/2, for the shape (Gelman, 2004). However, it is critical that the current experiment and the previous experiment from which one derives the hyperparameter for the current analysis are exchangeable, that is, that the two used the same population, covariates, and measurement instruments (e.g., scales) for all variables (see Miočević et al, 2020 for more on exchangeability). Thus, trauma research serves to benefit from specifying informative priors, as it may help build on previous knowledge and increase power to detect effects in small samples.…”

Perspectives on Bayesian inference and their implications for data analysis.

“…Operating from this perspective motivates specifying a prior distribution that reflects whatever is believed about the parameters before the data are observed. Primary sources for prior beliefs include judgments of subject matter experts (e.g., Abrams et al, 1994; van de Schoot et al, 2018; Zondervan-Zwijnenburg et al, 2017) and the results of past analyses (e.g., de Leeuw & Klugkist, 2012; Miočević et al, 2020). This is a broader lens on the prior specification to be updated, as a prior distribution is built to reflect substantive beliefs about the situation, possibly subject to constraints on what analyses are tolerated (Lee & Vanpaemel, 2018; Levy & Crawford, 2009; Levy & Mislevy, 2016; Martin & McDonald, 1975).…”

“…As an example of the type of research in line with Equation 3, Miočević et al (2020) conducted regression analyses from data collected at the Rothamsted Experimental Station in 1935 and 1936 to model the relationship between the dose of fertilizer and pounds of grain harvested. They first analyzed the 1935 data.…”