BACKGROUND In 2021, European Union registered over 365,000 excess deaths, with over 16,000 excess deaths in Portugal. The Portuguese Directorate-General of Health (DGS) has developed a deep neural network – AUTOCOD – that codifies primary causes of death by analyzing the free text in the physicians' death certificates (DC). Although the performance of AUTOCOD has already been demonstrated, it was not clear if this performance was the same over time, especially during excess mortality periods. OBJECTIVE Determine the sensitivity of AUTOCOD for classifying the underlying cause of death compared with manual coding to ascertain the specific causes of death, in periods of excess mortality. METHODS We included all the DC between 2016 and 2019. We evaluated the performance of AUTOCOD through a confusion matrix, comparing ICD-10 classifications of DC by AUTOCOD with those from the human coders at DGS (gold-standard). Next, we compared the periods without excess mortality with periods of excess, severe and extreme excess mortality. Lastly, we repeated the analyses for the three most common ICD-10 chapters, targeting classification at the block level. RESULTS AUTOCOD showed high sensitivity (≥0.75) for ten ICD-10 chapters studied, with values above 0.90 for the more prevalent chapters (II – neoplasms; IX – diseases of the circulatory system; X – diseases of the respiratory system). These high sensitivity values show no significant differences when comparing the periods without excess mortality with periods of excess, severe and extreme excess mortality. When considering the ICD-10 block classification of the three most common ICD-10 chapters, AUTOCOD again performed well, showing high sensitivity (≥0.75) for 13 ICD-10 blocks, with no significant differences between periods without excess mortality and periods with excess mortality. CONCLUSIONS Our results suggest that even during periods of excess and extreme excess mortality, the performance of AUTOCOD is not affected by a potential loss in text quality due to pressure on health services. Thus, AUTOCOD can be reliably used for real-time specific-cause mortality surveillance even in extreme excess mortality.

Background In 2021, the European Union reported >270,000 excess deaths, including >16,000 in Portugal. The Portuguese Directorate-General of Health developed a deep neural network, AUTOCOD, which determines the primary causes of death by analyzing the free text of physicians’ death certificates (DCs). Although AUTOCOD’s performance has been established, it remains unclear whether its performance remains consistent over time, particularly during periods of excess mortality. Objective This study aims to assess the sensitivity and other performance metrics of AUTOCOD in classifying underlying causes of death compared with manual coding to identify specific causes of death during periods of excess mortality. Methods We included all DCs between 2016 and 2019. AUTOCOD’s performance was evaluated by calculating various performance metrics, such as sensitivity, specificity, positive predictive value (PPV), and F1-score, using a confusion matrix. This compared International Statistical Classification of Diseases and Health-Related Problems, 10th Revision (ICD-10), classifications of DCs by AUTOCOD with those by human coders at the Directorate-General of Health (gold standard). Subsequently, we compared periods without excess mortality with periods of excess, severe, and extreme excess mortality. We defined excess mortality as 2 consecutive days with a Z score above the 95% baseline limit, severe excess mortality as 2 consecutive days with a Z score >4 SDs, and extreme excess mortality as 2 consecutive days with a Z score >6 SDs. Finally, we repeated the analyses for the 3 most common ICD-10 chapters focusing on block-level classification. Results We analyzed a large data set comprising 330,098 DCs classified by both human coders and AUTOCOD. AUTOCOD demonstrated high sensitivity (≥0.75) for 10 ICD-10 chapters examined, with values surpassing 0.90 for the more prevalent chapters (chapter II—“Neoplasms,” chapter IX—“Diseases of the circulatory system,” and chapter X—“Diseases of the respiratory system”), accounting for 67.69% (223,459/330,098) of all human-coded causes of death. No substantial differences were observed in these high-sensitivity values when comparing periods without excess mortality with periods of excess, severe, and extreme excess mortality. The same holds for specificity, which exceeded 0.96 for all chapters examined, and for PPV, which surpassed 0.75 in 9 chapters, including the more prevalent ones. When considering block classification within the 3 most common ICD-10 chapters, AUTOCOD maintained a high performance, demonstrating high sensitivity (≥0.75) for 13 ICD-10 blocks, high PPV for 9 blocks, and specificity of >0.98 in all blocks, with no significant differences between periods without excess mortality and those with excess mortality. Conclusions Our findings indicate that, during periods of excess and extreme excess mortality, AUTOCOD’s performance remains unaffected by potential text quality degradation because of pressure on health services. Consequently, AUTOCOD can be dependably used for real-time cause-specific mortality surveillance even in extreme excess mortality situations.