Sources of Error in the Coding of Questionnaire Data

Abstract

Errors made in the coding of a set of 1,209 relatively structured questionnaires were examined in an effort to discover characteristics of the data set and of the coding instructions which contributed to the occurrence of error. Items included in the questionnaire were ordered with respect to the complexity of the coding task involved. The number of errors per decision was found to vary directly with the level of complexity of the task. Closer examination of the types of decisions involved, including the computation of proportion agreement with criterion codes and validity coefficients, revealed sources of error within different levels of complexity. The use of non-mutually exclusive sets of coding categories was found to introduce error into the coding process, and it is suggested that this error is not reflected in resultant measures of reliability and validity. Decisions made in a series applying the same set of categories to a number of similar responses within an item were found to effect lower validity coefficients for later decisions in the series. ecent research has focused attention on the nature . and sources of error generated in the process of coding sociological data (Kammeyer and Roth, 1971; Crittenden and Hill, 1971;Crittenden, 1971). Though each investigation has dealt with a slightly different aspect of the coding process, there has been agreement that certain types of materials present the coder with a more complex task than other types of materials.

Coding In-depth Semistructured Interviews

“…There is considerable debate about whether clearly demarcated parts of the text, such as a sentence or paragraph, rather than ''units of meaning'' as defined by the coder are the appropriate units of analysis (Garrison et al 2006;Morrissey 1974). On one hand, the concern with using predefined blocks of text is that it may not accurately reflect the meaning as intended by the respondent.…”

“…It is not ordinarily recommended that intercoder reliability be calculated simply as the percentage of agreement among coders-the so-called proportion agreement method (Morrissey 1974)-because this does not take into consideration the possibility that coders might agree occasionally by chance (Bernard 2000:459-61). Chance may inflate agreement percentages, especially with only two coders and when they have only a few codes (Grayson and Rust 2001).…”

The Effect of Technology-Enhanced Classrooms in Middle School Education

“…inter-coder reliability was obtained, no modifications were needed for consistency (Miles & Huberman, 1994) since, as Krippendorff (2004) and Morrissey (1974) stated, two or more separate coders are needed for more than 90% of inter-coder agreement.…”

A Study of Coder Variability

“…Although such errors may be studied fairly easily, they have received little attention in the literature. Yet the few studies that have been reported have shown that when complex or judgemental codings are involved coding errors can seriously impair the quality of the resulting data (Woodward and Franzen, 1948;Durbin and Stuart, 1954;Sussman and Haug, 1967;Crittenden and Hill, 1971;Kammeyer and Roth, 1971; U.S. Bureau of the Census, 1972, 1974Duncan and Evers, 1975). In view of this situation it was decided to carry out a small-scale experiment, similar in design to that of Durbin and Stuart (1954), to examine the coding reliability achieved by professional survey organization coders in making judgemental codings of answers to open questions asked on a social survey questionnaire.…”