Evaluator’s alignment as an important indicator of reliability of assessment procedure for recognizing the good practice in public health: piloting newly developed criteria.

Abstract

Background The Slovenian “Criteria for assessing public health interventions for the purpose of identifying and selecting good practices” is a tool designed to identify and select good practices in public health. Public health professionals used the criteria to evaluate interventions and assign scores based on three levels: Exclusion, Core, and Additional Criteria. The assessment took place from 2021 to 2022. The individual scores were compiled and a panel discussion was held to reach a final score. The interrater agreement was measured using percent overall agreement and Fleiss’s kappa coefficient. Results The study aimed to test the consistency of the criteria, and despite being the first time the criteria were used, a moderate interrater agreement was achieved. The lowest agreement was observed for the effectiveness and efficiency sub-criteria group. Challenges identified with the scoring process were due to the descriptive 1 to 5 scale and the varying specificity of the criteria. The results showed that studying consistency between evaluators can highlight areas for improvement or adjustment in assessment criteria and enhance the quality of the assessment instrument. Conclusions This study aimed to evaluate the efficacy of the Slovenian “Criteria for assessing public health interventions for the purpose of identifying and selecting good practices”, which was found to offer valuable insights for various stakeholders. To enhance the quality of best practice assessments, the study suggests exploring opportunities for involving policy makers in the assessment process. Such collaboration between researchers, practitioners, and decision-makers can lead to significant improvements in the production and application of public health research.