Researching the Reliability of Accreditation Survey Teams: Lessons Learnt When Things Went Awry

Abstract

Accreditation of health organisations, occurring in over 70 countries, is predicated upon the reliability of survey teams judgements, but we do not know the extent to which survey teams are reliable. To contribute evidence to this issue, we investigated the reliability of two survey teams simultaneously assessing an organisation. The setting was a large Australian teaching hospital, and data were derived from interviews, observations and survey documents. Participants were from four groups: hospital staff, accreditation agency personnel and surveyors, and research staff. Thematic analysis was employed to identify significant factors that influenced the study. The two survey teams ratings and recommendations demonstrated high levels of agreement. However, while a common understanding of the study existed, the research was compromised. There were difficulties enacting the study. Contrary to negotiated arrangements, the pressure of the study resulted in surveyors discussing evidence and their interpretation of standards. Uncontrollable circumstances (late changes of personnel), and unexpected events (a breakdown of working relationships), challenged the study. The twin lessons learnt are that a consistent survey outcome is likely to be reached when reliability of process and consistent application of standards are pursued, and research requires negotiating challenges and relationships.

The impact of national accreditation reform on survey reliability: a 2‐year investigation of survey coordinators’ perspectives

“…The vast majority of feedback data was clear and specific with respect to different elements in the accreditation process. Several iterations of working through the list, with ongoing discussions informed by the study team's qualitative research experience and prior work into reliability [1,2,12,19–21], produced the final categories and refined their descriptions. Each year, the reform process and use of the NSQHS Standards stimulated new specific concerns for coordinators, and consequently generated new questions within each category.…”

“…Within each category findings table, details are presented by year and, as different items arose each year, the details and number of items change. The broad pattern of responses was the focus of analysis; hence participants' responses were classified into three groups: agreement (7-10), neutral (5-6) and disagreement (1)(2)(3)(4). This approach enabled analysis of coordinators' collective experience of significant reliability issues and consideration of their continued relevancy.…”

Understanding the Evolution of Health Service Accreditation Research: A Literature Review From 2012 to 2015

“…A research, developed to investigate the reliability of a survey on hospital staff judgment in a large Australian teaching hospital, was inconclusive (GREENFIELD et al, 2013b). The result was compromised by events beyond the study settings, interactions of surveyors, organizational influences on the surveyor and relationships between researcher and participants.…”

“…Nine studies discussed the survey and surveying issues (BRAITHWAITE et al, 2012;GREENFIELD et al, 2012aGREENFIELD et al, , 2012bGREENFIELD et al, , 2013bGREENFIELD et al, , 2015AGHEORGHIESEI;ILIESCU;GAVRILOVICI, 2013;CHUANG;HOWLEY;HANCOCK, 2013;AGHEORGHIESEI et al, 2014;TOWERS;CLARK, 2014), by assessing the function and reliability of accreditation surveys and surveyors. The results suggest that recent researches have focused more on surveying issues.…”

Avaliação do impacto dos programas de acreditação nas instituições de saúde brasileiras: contribuições de um estudo bibliométrico e de uma pesquisa de avaliação.

“…Um estudo desenvolvido para investigar a confiabilidade de uma pesquisa no julgamento pessoal da equipe, em um grande hospital de ensino da Austrália, foi inconclusiva (GREENFIELD et al, 2013b (BRAITHWAITE et al, 2012;SAX;MARX, 2013;SHAW et al, 2013;TABRIZI;GHARIBI;PIRAHARY, 2013;WOODHEAD, 2013;GREENFIELD et al, 2014aGREENFIELD et al, , 2014bGREENFIELD et al, , 2015HANDAYANI et al, 2015). A maior parte deles destacou as preocupações das partes interessadas (stakeholders), tais como o governo, as agências e a comunidade, além do processo colaborativo.…”