Reply to comments by Jackes: Interobserver error and goodness‐of‐fit tests in paleodemography

Abstract

Mary Jackes' comments on our previous article (Konigsberg and Frankenberg, 2002) contain the important lesson, that either: 1) good methods applied to good data can produce bad results if important assumptions are violated, or 2) good methods applied to bad data can produce bad results (the "garbage in, garbage out" maxim). Jackes appears to incline toward the first explanation in her comment, while we show in this reply that the second possibility is more parsimonious. Jackes' conclusion that her "test has shown that iterative proportional fitting does not provide an accurate mean adult age at death" also suggests that she is arguing that the specific method is bad.Our first issue with Jackes' comments is her statement that "a floating point error makes it impossible to iterate 17 ages for six stages." We noted in our article that when the number of age classes is greater than the number of stages, the "iterative proportional fitting procedure" (IPFP) is inappropriate and should not be used. It is often possible in such cases to run an IPFP program and produce output, but when there are more ages than stages, that output is uninterpretable gibberish. Jackes notes in the first paragraph of her comment that "the number of age groups is constrained by the number of morphological stages, a disadvantage of IPFP," and yet she then goes on to run an IPFP using 16 age categories and only six pubic symphyseal stages. It is not simply our "opinion" (quoting Jackes) that such an application is improper; using the method in this way is statistically invalid (see Clark 1981, p. 299; Kimura and Chikuni, 1987, p. 28; Hoenig and Heisey, 1987, p. 235).Our second issue with Jackes' argument is that she does not employ a goodness-of-fit statistic to evaluate the IPFP when applied properly (her six stage and five age classes example). One should calculate some form of goodness-of-fit statistic when fitting models in order to both identify the more parsimonious models and assure that the model adequately reproduces the observed data (counts in stages). To quote from Hoenig and Heisey (1987, p. 235The main assumption of our procedure is described by equation (1); the misclassification rates are the same for samples 1 and 2. This can be tested with either a likelihood ratio or Pearson chi-square goodness-of-fit test with J Ϫ I degrees of freedom.