Patterns of Interobserver Error in the Scoring of Entheseal Changes

Abstract

Although the interpretation of entheses has undergone considerable discussion recently, the role of interobserver error, especially in comparative contexts, has been only sporadically addressed. Using standards developed by Hawkey and Merbs, currently the most widely used system, rates of reproducibility were evaluated in two prehistoric North American skeletal series. Eight observers of varying experience levels scored 17 long bone entheses, representing both fibrous and fibrocartilaginous attachment types, on 58 individuals. Results showed rates of reproducibility to be only marginally higher than what would be expected by chance alone. Observer experience level did not appear to be a factor nor was attachment type. As might be predicted, those entheses enjoying the highest rates of reproducibility exhibited relatively smooth attachment morphology and/or less defined boundaries whereas those with the lowest rates displayed the greatest range of surface morphology expression. Possible explanations for the levels of interobserver error observed include difficulties in reducing the highly variable enthesis morphology to a few discrete categories, categories that encompass too many criteria, and use of vague terminology in describing morphological features. Consequently, comparison of data across studies by different observers, especially those not trained by the developer of a given scoring method, must be undertaken with great caution. Copyright © 2012 John Wiley & Sons, Ltd.

Subsistence strategy changes: The evidence of entheseal changes

“…It is unlikely that all those researchers who have recorded agricultural populations routinely score lower than those recording hunter-gatherers, but inter-observer error must be considered. The original report (Hawkey and Merbs, 1995) presented low intra-and inter-observer error, but a recent study (Davis et al, 2013) found higher error rates. These are likely to be caused by the difficulty of using photographs to learn a recording method (Davis et al, 2013).…”

Upper limb entheseal change with the transition to agriculture in the southeastern United States: A view from Moundville and the central Tombigbee River valley

“…Villotte et al (2010) previously scored six primary FC attachments based on their association with skeletal modification in the medical literature. These have been shown to have characteristic remodeling on the entheseal surfaces (Villotte, 2006), with claims of high replicability (cf., Davis et al, 2012). Yet, response to external loadings is complex and poorly understood, and traditional visual scoring methods may not encompass the full range of human variation, which recently three-dimensional modeling is only beginning to show (Zumwalt, 2006:445).…”

“…To these we would add the limitations presented in our ability to reliably score only fibrocartilaginous muscle attachments (cf., Davis et al, 2012;Villotte et al, 2010). If we cannot reliably score large fibrous muscles, this places severe limitations on patterns of interpretation.…”

“…Limitations include the confounding effects of age, sex, and body size on the development of attachment sites (Milella et al, 2012;Niinimäki, 2011;Weiss, 2003Weiss, , 2004Weiss, , 2007 and difficulties in confirming specific actions from muscles in experimental activity studies on Ovis aries (Zumwalt, 2005(Zumwalt, , 2006. Several researchers have noted that relationships between entheseal change, hormones, disease, and genetics remain poorly understood, making interpretations difficult (Jurmain, 1999;Niinimäki, 2011;Zumwalt, 2006), and problems with existing scoring standards have been a pressing concern (Davis et al, 2012;Meyer et al, 2011;Villotte, 2006;Villotte et al, 2010).…”

Technical note: Quantifying size and shape of entheses

“…Inter-observer error, in contrast, was higher. This has also been found to be a problem for visual recording methods Davis et al, 2012b). The new 'Coimbra method' for recording entheses reported a low of 65.3% agreement and a high of 78.1% agreement between scorers, all of whom had considerable experience of recording ECs and were involved in developing the method .…”