Knowledge of Problem Solving (KOPS) Scale: Design and Evaluation of a Digitally Administered Competence Measure for a Common Practice Element in Task-Shared Youth Mental Health Interventions

Abstract

There are few reliable and feasible quality assurance methods to support scaling up of psychological interventions delivered by non-specialist providers. This paper reports on the phased development and validation of a digitally administered Knowledge of Problem Solving (KOPS) measure to assess competencies associated with a “task-shared” problem-solving intervention for adolescents with diverse mental health problems in India. Phase 1 established key competencies required to deliver the intervention, followed by item generation for a corresponding knowledge-based competency measure that could be administered efficiently through e-learning systems. In phase 2, items were refined based on responses from an “experienced” reference sample comprising 17 existing counsellors with direct experience of the problem-solving intervention, and a “novice” sample with 14 untrained university students and NGO staff. In phase 3, we evaluated two parallel versions of the measure in a validation sample (N = 277) drawn from universities and NGOs. The resulting 17-item measure was structured around a hypothetical case, followed by multiple-choice questions that asked about the most appropriate response to a practice-based scenario. The difficulty level of the test items was well matched to the ability level of participants (i.e. most items being of moderate difficulty and few items being easy or difficult). Only one item showed a negative discrimination index and was removed from the 17-item forms. The final 16-item version of the KOPS measure provides a scalable digital method to assess key psychotherapeutic competencies among non-specialists, particularly in relation to a transdiagnostic problem-solving intervention. Similar formats could be deployed more widely alongside e-learning programmes to expand the global workforce capable of delivering evidence-based psychological interventions.

Developing knowledge-based psychotherapeutic competencies in non-specialist providers: A pre-post study with a nested randomised controlled trial of a coach-supported versus self-guided digital training course for a problem-solving psychological intervention in India

“…As such, the assessed items correspond to "knowing" and "knowing how" rather than "showing how" or "doing" in the nomenclature of Miller's (1990) hierarchy of clinical competency. Development and validation of the measure have been described elsewhere (Mathur et al, 2023a). The measure comprised five session vignettes for a hypothetical case, with each vignette followed by 3-4 multiple-choice questions that asked about the most appropriate response to a practice-based scenario.…”