The UNICEF/Washington Group Child Functioning Module—Accuracy, Inter-Rater Reliability and Cut-Off Level for Disability Disaggregation of Fiji’s Education Management Information System

Abstract

This paper explores the validity (sensitivity and specificity) of different cut-off levels of the UNICEF/Washington Group Child Functioning Module (CFM) and the inter-rater reliability between teachers and parents as proxy respondents, for disaggregating Fiji’s education management information system (EMIS) by disability. The method used was a cross-sectional diagnostic accuracy study comparing CFM items to standard clinical assessments for 472 primary school aged students in Fiji. Whilst previous domain-specific results showed “good” to “excellent” accuracy of the CFM domains seeing, hearing, walking and speaking, newer analysis shows only “fair” to “poor” accuracy of the cognitive domains (learning, remembering and focusing attention) and “fair” of the overall CFM (area under the Receiver Operating Characteristic curve: 0.763 parent responses, 0.786 teacher responses). Severe impairments are reported relatively evenly across CFM response categories “some difficulty”, “a lot of difficulty” and “cannot do at all”. Most moderate impairments are reported as “some difficulty”. The CFM provides a core component of data required for disaggregating Fiji’s EMIS by disability. However, choice of cut-off level and mixture of impairment severity reported across response categories are challenges. The CFM alone is not accurate enough to determine funding eligibility. For identifying children with disabilities, the CFM should be part of a broader data collection including learning and support needs data and undertaking eligibility verification visits.

Intra-Rater Test-Retest Reliability of a Modified Child Functioning Module, Self-Report Version

“…The items on seeing, hearing, remembering things, controlling behavior, and getting friends could be interpreted to have had large agreement when the first level cut-off (at least some difficulties) was administered. These results partly contrast with the earlier evidence from the draft reliability tests of the CFM, whereby the levels agreements between parents and teachers were poor when reporting children who have at least some difficulty in getting friends [ 35 ]. Further evidence is needed that can be explained through triangulating the data from three main sources, the pupil, parents, and teacher before conclusions about these differing items can be made.…”

“…However, the level of agreement on difficulties with self-care, concentrating on things the child enjoys, and having changes to the routine were only fair. Similar problems with the item on difficulties with self-care were reported in an inter-rater reliability study between parents and teachers, which found poor agreement [ 35 ]. This would suggest that these items may have different meanings at different times of survey completion, and may need to be interpreted with caution [ 21 ].…”

“…The use of the CFM for self-reported disabilities is, overall, an acceptable measure. This is an important finding because previous work on these items has been based on proxy reporting [ 13 , 35 ] and there is a need to include self-reported disabilities in national health surveys [ 5 ]. Young adolescence is an important time for reflecting on personal growth and these changes must be monitored [ 19 ].…”

“…The item on concentrating was created as an extension of the Washington Group short set of six items, whereby one of item was related to “difficulties in remembering and concentration” [ 39 ]. The separation of items were not featured in an earlier draft reliability study (collected in 2015) of the CFM in a special education setting [ 35 ], indicating the possibility of a low level of evidence for the item. One observation of the items, as a whole, is that there are more child-related domains in the CFM and it is assumed they are uniform across the ages of 5–18 years old [ 15 ].…”

Child functioning and disability in children living with human immunodeficiency virus in a semi-rural healthcare setting in South Africa

“…For the purpose of this study, a child was considered to be experiencing disability at the level of ‘some difficulty’ with carrying out functional activities. 18 The tool has shown good diagnostic accuracy 19 in international research, and studies suggest using the cut off ‘some difficulty’ to identify children at risk of disability and then refer to a medical practitioner for an objective assessment. 18 , 19 …”