Computational methods for Bayesian semiparametric Item Response Theory models

Abstract

Item response theory (IRT) models are widely used to obtain interpretable inference when analyzing data from questionnaires, scaling binary responses into continuous constructs. Typically, these models rely on a normality assumption for the latent trait characterizing individuals in the population under study. However, this assumption can be unrealistic and lead to biased results. We relax the normality assumption by considering a flexible Dirichlet Process mixture model as a nonparametric prior on the distribution of the individual latent traits. Although this approach has been considered in the literature before, there is a lack of comprehensive studies of such models or general software tools. To fill this gap, we show how the NIMBLE framework for hierarchical statistical modeling enables the use of flexible priors on the latent trait distribution, specifically illustrating the use of Dirichlet Process mixtures in two-parameter logistic (2PL) IRT models. We study how different sets of constraints can lead to model identifiability and give guidance on eliciting prior distributions. Using both simulated and real-world data, we conduct an in-depth study of Markov chain Monte Carlo posterior sampling efficiency for several sampling strategies. These strategies consider various combinations of model parameterizations, identifiability constraints and sampling algorithms. In the simulated scenarios, we find that some choices of parameterization and identifiability constraints can yield order-of-magnitude differences in sampling efficiency compared to others. We also find that there is little inferential penalty in using

Comparing the MCMC Efficiency of JAGS and Stan for the Multi-Level Intercept-Only Model in the Covariance- and Mean-Based and Classic Parametrization

“…We used a high number of iterations to achieve sensibly sized absolute runtimes and to obtain reliable ESS estimates [38]. We assume that the MCMC efficiency (ESS/s) is constant over the course of sampling.…”

“…We limited our study to two popular Bayesian software in psychological research, namely JAGS and Stan. Other Bayesain software packages, for instance, NIMBLE [10], PyMC3 [39], and LaplacesDemon [40], exist and have been the focus of research (e.g., [38,[41][42][43]). Future software comparisons should take these packages even more strongly into account.…”