An accurate and efficient identification of children with psychosocial problems by means of computerized adaptive testing

Abstract

BackgroundQuestionnaires used by health services to identify children with psychosocial problems are often rather short. The psychometric properties of such short questionnaires are mostly less than needed for an accurate distinction between children with and without problems. We aimed to assess whether a short Computerized Adaptive Test (CAT) can overcome the weaknesses of short written questionnaires when identifying children with psychosocial problems.MethodWe used a Dutch national data set obtained from parents of children invited for a routine health examination by Preventive Child Healthcare with 205 items on behavioral and emotional problems (n = 2,041, response 84%). In a random subsample we determined which items met the requirements of an Item Response Theory (IRT) model to a sufficient degree. Using those items, item parameters necessary for a CAT were calculated and a cut-off point was defined. In the remaining subsample we determined the validity and efficiency of a Computerized Adaptive Test using simulation techniques, with current treatment status and a clinical score on the Total Problem Scale (TPS) of the Child Behavior Checklist as criteria.ResultsOut of 205 items available 190 sufficiently met the criteria of the underlying IRT model. For 90% of the children a score above or below cut-off point could be determined with 95% accuracy. The mean number of items needed to achieve this was 12. Sensitivity and specificity with the TPS as a criterion were 0.89 and 0.91, respectively.ConclusionAn IRT-based CAT is a very promising option for the identification of psychosocial problems in children, as it can lead to an efficient, yet high-quality identification. The results of our simulation study need to be replicated in a real-life administration of this CAT.

Computerized adaptive testing to screen children for emotional and behavioral problems by preventive child healthcare

“…The mostly good validity indices for the Psycat in routine PCH confirm the findings of the previous simulation study regarding the Psycat of Vogels et al [19] and other (simulation) studies regarding CATs aimed to measure quality of life [16,18] or mental health [17] in the field of pediatric mental health care. Gardner's CAT assessed the same construct (mental health in children) as the Psycat.…”

“…The Psycat was an efficient procedure, requiring a mean number of 16 items to correctly identify a child, which is however four more items than in the simulation study [19]. This discrepancy can be explained in two ways.…”

“…The item pool of the PSYCAT was based on the items from four questionnaires on EBP, i.e. the Pediatric Symptom Checklist (PSC), the SDQ, the PSYBOBA, and the CBCL [19,20]. We separately determined Psycat outcomes for overall EBP, and for internalizing, externalizing, and hyperactivity problems.…”

“…Computerized Adaptive Testing (CAT) is a promising alternative for the identification of EBP in community settings as it can improve the quality of early detection while remaining brief [15][16][17][18][19]. In a CAT, a computer selects the items to be asked based on the parent's or child's previous responses.…”

“…Gardner et al (2002) [17] showed that a CAT based on the Pediatric Symptom Checklist (PSC) for parents in a simulation study can be used to measure mental health in children. Vogels et al (2011) [19] developed an Internet application -Psycatthat allows parents to use the CAT procedure to support identification of EBP in their 10-11-year-olds. In a simulation study, this Psycat was a highly sensitive and specific procedure for the identification of children with a clinical total problem score on the Child Behavior Checklist (CBCL), with an average 12 items needed to identify children with EBP [19].…”