A Study of Coder Variability

Kalton, Graham; Stowel, Richard

doi:10.2307/2347199

Cited by 11 publications

(4 citation statements)

References 17 publications

Supporting

Mentioning

Contrasting

Unclassified

Order By: Relevance

“…There are many measures for examining agreement across raters, differing in whether the measure accounts for agreements occurring by chance, whether the ratings are assumed to be nominal or ordinal, and whether items and raters are assumed to be fixed or randomly selected (Schrout and Fleiss 1979;Banerjee et al 1999). As such, three measures were used to indicate the degree of agreement across the reviewers in their ratings-the proportion of all two-way combinations of reviewers who provide exactly the same rating across all of the questions (the match rate), a multiple coder kappa (Kalton and Stowell 1979), and an intraclass correlation coefficient (ICC). The match rate is intuitive but does not account for chance agreement.…”

Section: Discussionmentioning

confidence: 99%

An Examination of Questionnaire Evaluation by Expert Reviewers

2010

View full text Add to dashboard Cite

Expert reviews are frequently used as a questionnaire evaluation method but have received little empirical attention. Questions from two surveys are evaluated by six expert reviewers using a standardized evaluation form. Each of the questions has validation data available from records. Large inconsistencies in ratings across the six experts are found. Despite the lack of reliability, the average expert ratings successfully identify questions that had higher item nonresponse rates and higher levels of inaccurate reporting. This article provides empirical evidence that experts are able to discern questions that manifest data quality problems, even if individual experts vary in what they rate as being problematic. Compared to a publicly available computerized question evaluation tool, ratings by the human experts positively predict questions with data quality problems, whereas the computerized tool varies in success in identifying these questions. These results indicate that expert reviews have value in identifying question problems that result in lower survey data quality. 295

show abstract

Section: Discussionmentioning

confidence: 99%

An Examination of Questionnaire Evaluation by Expert Reviewers

2010

View full text Add to dashboard Cite

show abstract

“…We each coded the responses independently and then reconciled all differences. Computing interrater reliability with multiple coders and nonmutually exclusive codes was made possible by amending the formula provided by Kalton and Stowell (1979). For objection, we had 73 percent agreement and a kappa score of .65.…”

Section: Note: All Examples In This Table Are Drawn From Responses To Vignettes In Which the Dependent Variable Was Problematicmentioning

confidence: 99%

Diversity in Everyday Research Practice

Leahey

Entwisle

Einaudi

2003

Sociological Methods & Research

View full text Add to dashboard Cite

How should social science researchers deal with data inaccuracies? This article uses Web-based survey data collected from faculty members in three social science disciplines to document variation in views about data editing. Through an analysis of qualitative responses to a hypothetical vignette, the authors demonstrate that a wide range of opinion surrounds the “proper” use of data. Reactions are to some extent contingent on discipline and experience with different types of data and data collection methods. They also depend on characteristics of the data-editing situation—for example, whether the problem is with an independent or dependent variable. Even taking these social and situational factors into account, however, there is still substantial diversity in vignette responses. Normative standards that pervade other aspects of the research process have not yet emerged for data editing.

show abstract

“…All codes and responses were checked for accuracy by members of the research team and subsequently given a numeric value to aid quantitative analysis. Intra-coder reliability was employed in this study to enhance the coding process of qualitative data ( 31 ). A subset of the data was coded at different time points, allowing for the assessment of agreement between the coder’s coding decisions.…”

Section: Methodsmentioning

confidence: 99%

Facilitators and barriers to physical activity among English adolescents in secondary schools: a mixed method study

Moore,

Vernon,

Gregory

et al. 2023

Front. Public Health

View full text Add to dashboard Cite

BackgroundIt is evident that physical activity (PA) programmes implemented in schools were not effective in improving PA behaviours among adolescents. This study investigated students’ perceptions of barriers to PA among inactive English adolescents in secondary schools based on the Capability, Opportunity, Motivation, and Behaviour (COM-B) model, the Behaviour Change Wheel (BCW), and Theoretical Domains Framework (TDF). The study compared barriers faced by inactive and active groups participating in sports and PA in secondary schools to identify sources of behaviour contributing to inactivity.MethodsA pre-intervention online survey was distributed to affiliated schools by 233 Teaching Schools Alliances (TSAs) as part of the monitoring and evaluation of the Secondary Teacher Training study. Data were cross-tabulated to analyse activity levels and behavioural barriers for active and inactive groups, using the COM-B domains. The research team followed a seven-step process to categorise barriers based on their relevant domain in the TDF mapped to the COM-B.ResultsThe findings were derived from one of the most extensive surveys of adolescents ever undertaken involving 200,623 active and 8,231 inactive respondents. The study identified 52 barriers and 68 behaviours that prevent adolescents from participating in PA. Psychological and social barriers were found to affect all activity levels, genders, and ethnic groups, with a lack of confidence and self-consciousness being the most prevalent. Certain demographic groups, such as those from minority ethnic groups and disabled individuals, were found to be overrepresented among inactive populations. The finding of the study indicated that there were common barriers that affected both inactive and active groups, with further similarity when examining barriers between active and inactive girls. The study also found that girls were more likely to experience the main barriers compared to boys, while inactive boys were more likely to encounter these barriers compared to active boys. The findings suggest that common barriers could be addressed across the population, while recognising some differences in demographics, and the need to provide personalised support. Targeted interventions are also suggested for all girls and inactive boys.ConclusionThis study highlights the range of barriers that impact adolescents and provides insight into potential mechanisms for behaviour change, including intervention functions, policy categories, and evidence-based behaviour change tools. The study highlights the need for further research to address the barriers to PA among adolescents, particularly those who are inactive. Utilising the findings of this study, future research should investigate the effectiveness of novel digital exercise interventions and policies in increasing PA levels among children and adolescents. Complex digital exercise interventions, including conversational AI solutions, could provide personalised tools to identify and revolutionise support around the multitude of barriers that impact adolescents globally.“For the purpose of open access, the author has applied a Creative Commons Attribution (CC BY) licence to any Author Accepted Manuscript version arising from this submission.”

show abstract

A Study of Coder Variability

Cited by 11 publications

References 17 publications

An Examination of Questionnaire Evaluation by Expert Reviewers

An Examination of Questionnaire Evaluation by Expert Reviewers

Diversity in Everyday Research Practice

Facilitators and barriers to physical activity among English adolescents in secondary schools: a mixed method study

Contact Info

Product

Resources

About