Blind Testing of Faunal Identification Protocols: A Case Study with North American Artiodactyl Stylohyoids

Abstract

Taxonomic identification of archaeofauna relies on techniques and anatomical traits that should be valid, reliable, and usable, but which are rarely tested. Identification protocols (techniques and anatomical traits), particularly those used to distinguish taxa of similar size and morphology, should be rigorously tested to ensure a solid interpretive foundation. Blind testing of a protocol for identifying stylohyoid bones of North American artiodactyls was performed by three analysts who independently employed the protocol to identify 77 anatomically complete specimens of known taxonomic identity, representing 54 individuals and 11 species. Identifications were identical in 89% of cases and in conflict in 3% of cases. The remainder involved differences in resolution; two analysts identified specimens to species, whereas the third identified specimens to more general taxonomic groups. Inter-analyst variability in identification was a result of differences in protocol application. Identifications were consistent with known taxon in 92%–96% of cases. Results indicate that the protocol is valid, reliable, and usable, and it can be applied to archaeological specimens with confidence. Testing of other identification criteria employed by zooarchaeologists is encouraged.

Identifying marsupials from Australian archaeological sites: current methodological challenges and opportunities in zooarchaeological practice

“…Further, inter-operator subjectivity in undertaking zooarchaeological analysis has been demonstrated to produce variable data even between experienced analysts. Participants in experimental blind tests conducted by Gobalet ( 2001), Morin et al (2017), Lau and Kansa (2018) and Lubinski et al (2020) not only disagreed on taxonomic identifications of specimens but also varied in their willingness to discriminate to family versus species level; 138 Identifying marsupials from Australian archaeological sites NISP and MNI; the designation of size classes; the percentage of the assemblage deemed identifiable and, in rare cases, even disagreed on the skeletal element. The variation in data produced in blind tests further highlights the need for methodological transparency in zooarchaeological data production.…”

“…(2017), Lau and Kansa (2018) and Lubinski et al . (2020) not only disagreed on taxonomic identifications of specimens but also varied in their willingness to discriminate to family versus species level; NISP and MNI; the designation of size classes; the percentage of the assemblage deemed identifiable and, in rare cases, even disagreed on the skeletal element. The variation in data produced in blind tests further highlights the need for methodological transparency in zooarchaeological data production.…”

“…Traditional qualitative protocols use descriptive guides, often pictorial and supported by descriptions of diagnostic criteria. These protocols rely on subjective terminology to describe bone morphology such as 'more square' or 'more ovoid', but can be tested for replicability and robustness across multiple practitioners and datasets, and the efficacy of each diagnostic characteristic continually evaluated over time (Lambrides & Weisler 2015;Lubinski et al 2020;Zeder & Lapham 2010). Quantitative identification protocols generally rely on the collection of morphometric data, either in two or three dimensions.…”

Practice makes perfect? Inter‐analyst variation in the identification of fish remains from archaeological sites

“…Typically, this occurs in a hands‐on learning environment with some assistance from texts (e.g., Beisaw, 2013). Most professional zooarchaeologists make identifications through comparing the unknown specimen with reference specimens and employing morphological and metric characteristics combined with application of knowledge of diagnostic traits (Lubinski et al, 2020; Lyman, 2019), ideally using physical reference collections including multiple examples of different ages, sizes, and sexes (e.g., Baker & Worley, 2019) and possibly including established protocols for identifications (e.g., Lubinski & Hale, 2018). Online digital comparative databases are increasingly available (e.g., Betts et al, 2011) and may be used as aids in identification.…”

Computed Tomography in Medicolegal Death Investigation: A Critical Review

“…For example, if a particular study is repeated a number of times, a low variability in results would indicate a high level of precision. Accuracy will refer to how close results are to an objective "truth" (14). An example of high accuracy would mean that the average results from a particular sample would be very close to the true average of a population.…”