Occupation Coding of Job Titles: Iterative Development of an Automated Coding Algorithm for the Canadian National Occupation Classification (ACA-NOC)

Abstract

Background In many research studies, the identification of social determinants is an important activity, in particular, information about occupations is frequently added to existing patient data. Such information is usually solicited during interviews with open-ended questions such as “What is your job?” and “What industry sector do you work in?” Before being able to use this information for further analysis, the responses need to be categorized using a coding system, such as the Canadian National Occupational Classification (NOC). Manual coding is the usual method, which is a time-consuming and error-prone activity, suitable for automation. Objective This study aims to facilitate automated coding by introducing a rigorous algorithm that will be able to identify the NOC (2016) codes using only job title and industry information as input. Using manually coded data sets, we sought to benchmark and iteratively improve the performance of the algorithm. Methods We developed the ACA-NOC algorithm based on the NOC (2016), which allowed users to match NOC codes with job and industry titles. We employed several different search strategies in the ACA-NOC algorithm to find the best match, including exact search, minor exact search, like search, near (same order) search, near (different order) search, any search, and weak match search. In addition, a filtering step based on the hierarchical structure of the NOC data was applied to the algorithm to select the best matching codes. Results The ACA-NOC was applied to over 500 manually coded job and industry titles. The accuracy rate at the four-digit NOC code level was 58.7% (332/566) and improved when broader job categories were considered (65.0% at the three-digit NOC code level, 72.3% at the two-digit NOC code level, and 81.6% at the one-digit NOC code level). Conclusions The ACA-NOC is a rigorous algorithm for automatically coding the Canadian NOC system and has been evaluated using real-world data. It allows researchers to code moderate-sized data sets with occupation in a timely and cost-efficient manner such that further analytics are possible. Initial assessments indicate that it has state-of-the-art performance and is readily extensible upon further benchmarking on larger data sets.

Automated Coding of Job Descriptions From a General Population Study: Overview of Existing Tools, Their Application and Comparison

“…Some algorithms can also output justifications of choices or lists of multiple candidates which can be verified by manual coders. For example, AUTONOC has also recently introduced a portal for study managers to email study participants each a unique URL to self-code their occupation and select from candidates’ occupations provided by the algorithm ( Bao et al , 2020 ; Garcia et al , 2021).…”

“…Outputs in NOC 2016 were used for this analysis. ENENOC was benchmarked against a previous natural language processing (NLP) algorithm called ACA-NOC ( Bao et al , 2020 ) which was also developed for auto-coding to the Canadian National Occupational Classification. ENENOC showed improved performance on a curated test set (Garcia et al , 2021) and was selected for use in this study.…”

Automated occupation coding with hierarchical features: a data-centric approach to classification with pre-trained language models

“…The study of Bao et al [18] introduced a strict algorithm that will be able to identify the NOC (2016) codes using job titles and industry information as input exclusively. The ACA-NOC was applied to over 500 manually-coded job and industry titles.…”

Artificial intelligence exceeds humans in epidemiological job coding

“…Tools have been developed to support manual assignment of occupational codes to job descriptions 13 , 14 . Additionally, tools have been developed that enable completely automatic coding, achieving a prediction accuracy ranging 15–64% on the highest coding level with an out-of-distribution accuracy ranging 17–26% (see Table 1 ) 10 , 15 – 20 . Although some coding tools achieve a human-coder level prediction accuracy, their exposure assessment accuracy may be lower.…”