Three Methods for Occupation Coding Based on Statistical Learning

Abstract

Occupation coding, an important task in official statistics, refers to coding a respondent's text answer into one of many hundreds of occupation codes. To date, occupation coding is still at least partially conducted manually, at great expense. We propose three methods for automatic coding: combining separate models for the detailed occupation codes and for aggregate occupation codes, a hybrid method that combines a duplicate-based approach with a statistical learning algorithm, and a modified nearest neighbor approach. Using data from the German General Social Survey (ALLBUS), we show that the proposed methods improve on both the coding accuracy of the underlying statistical learning algorithm and the coding accuracy of duplicates where duplicates exist. Further, we find defining duplicates based on ngram variables (a concept from text mining) is preferable to one based on exact string matches.

How Standardized is Occupational Coding? A Comparison of Results from Different Coding Agencies in Germany

“…One may think that the problems resulting from differences in rules may be overcome by applying completely automatic coding routines. Indeed, such methods have been proposed, for example, based on statistical learning (for a discussion of such approaches, see Gweon et al 2017). However, up to now these approaches are not able to code considerably more than half of the occupations, thus limiting their usability.…”

Occupation Coding During the Interview

“…Speizer and Buckley () for a review, Measure () and Gweon et al . ()). Software for computer‐assisted coding suggests possible categories to the coder to make human work more efficient.…”

“…Several researchers have proposed computer systems to automate the post-survey coding process (e.g. Speizer and Buckley (1998) for a review, Measure (2014) and Gweon et al (2017)). Software for computer-assisted coding suggests possible categories to the coder to make human work more efficient.…”

Transforming Naturally Occurring Text Data Into Economic Statistics: The Case of Online Job Vacancy Postings

“…Until recently, methods which existed to label vacancy text with official classifications were proprietary, limited in the number of searches, or did not make use of the job description field. While writing up our results we became aware of similar approaches being developed for the US (Atalay et al, 2017), Germany (Gweon et al, 2017) and for the International Labour Organisation occupational classification (Boselli et al, 2017a,b).…”