Assessing reproducibility in faunal analysis using blind tests: A case study from northwestern North America

Abstract

Zooarchaeologists have long recognized that assigning taxonomic identifications to animal remains is a subjective process, and recent studies have highlighted the need for data quality assurance standards in archaeofaunal research. Our study contributes to this growing interest in quality assurance by presenting simple quantitative methods for assessing reliability in analytic results through blind reanalysis of animal remains that we developed during analysis of fishbone from Tse-whit-zen, a large Native American village on the coast of Washington State, U.S.A. Given the large scale of the Tse-whitzen project-over 112,000 fish remains were documented by five different analysts over three yearsthere was a real possibility that inconsistencies in laboratory practices affected analytic results (e.g., number of identified specimens, taxa present and relative abundance, elements identified, burning frequencies). To evaluate the overall reliability of the Tse-whit-zen fishbone data, and the possibility of "protocol drift"-changes in how specimens and bone modifications were identified over the course of analysis-we reanalyzed samples of fish remains that were previously documented during three discrete stages (beginning, middle, and end) of the Tse-whit-zen project. The original data and the reanalysis results show close agreement in each stage, with only minor differences in the numbers of recorded specimens, taxonomic representation at family-and finer taxonomic levels, and body part representation assessed for a single taxonomic order. Identifying burning on bone was not very reliable. Reliability studies such as this are useful for highlighting ambiguous identification criteria (e.g., in taxonomic assignment, bone modification), and could stimulate dialog among researchers about ways to address such issues in future studies. Increasing the implementation of these, and other, widely applicable quality control methods should improve zooarchaeological data quality and stimulate further research on quality assurance in archaeology overall.

Ancient DNA analysis of Indigenous rockfish use on the Pacific Coast: Implications for marine conservation areas and fisheries management

“…One sample demonstrated non-specific amplification of human DNA after multiple failed amplifications and was excluded from analyses. Although morphological misidentifications were rare, these results demonstrate that ancient DNA can act as a valuable check to confirm morphological identifications of archaeological remains [ 51 , 52 ].…”

Blind Testing of Faunal Identification Protocols: A Case Study with North American Artiodactyl Stylohyoids

“…Like Wolverton (2013) before us, we hope to see a marked improvement in zooarchaeological quality control. Nims and Butler's (2017) recommendation for continuous evaluation of the identification protocol being used over the duration of a project is a superb additional suggestion to enhance the probability of correct taxonomic identifications and reduce inter-observer variation. Much of modern zooarchaeology rests on high-resolution taxonomic identifications.…”

“…Furthermore, there tends to be minimal description of the identification protocol used in a particular study (e.g., Baker and Shaffer 1999; Driver 1992, 2011; Emery 2004; Lupo 2011; Lyman 2005; Stewart 2005). Authors of recent zooarchaeology textbooks and articles state identification “methods” should be described and published, particularly listing the reference collections and published skeletal guides consulted (e.g., Beisaw 2013; Driver 1992; Gobalet 2001; Hesse and Wapnish 1985; LeFebvre and Sharpe 2018; Nims and Butler 2017; O'Connor 2000; Rea 1986; Reitz and Wing 2008), but the suggestion is not consistently made, even in textbooks (e.g., Davis 1987; Gifford-Gonzalez 2018; Hillson 1992; Rackham 1994). Some authors have urged analysts to publish the skeletal traits used to make identifications (e.g., Butler 2011; Butler and Lyman 1996; Driver 1982; Gobalet 2001; Graham and Semken 1987; Klein and Cruz-Uribe 1984; Lawrence 1973; Lyman 2005, 2011a; Wolverton 2013; Wolverton and Nagaoka 2017).…”

“…The first is the “stability” of results (Heale and Twycross 2015) across repeated testing, known as “test–retest” (Roberts et al 2006). A test–retest method has been suggested by zooarchaeologists (Nims and Butler 2017; Wolverton and Nagaoka 2017), but reidentifying zooarchaeological specimens identified a few weeks ago does not test validity or accuracy, because the true identities are unknown. The second aspect of reliability and validity is “equivalence,” or consistency of results across multiple respondents (Heale and Twycross 2015).…”

A refitting experiment on long bone identification

“…Identifying faunal remains to skeletal element, and where possible, to taxon plays a pivotal role in most archaeozoological interpretations (Gifford‐Gonzalez, 2018; Nims & Butler, 2017; Reitz & Wing, 2008). However, substantial evidence suggests that certain categories of skeletal parts are easier to identify than others, a finding that affects both the accuracy and reproducibility of archaeozoological inferences (Hudson, 1990; Morin et al, 2017a, 2017b; Pickering et al, 2006).…”