HMORN-Selected Abstracts and those from people with comorbidities (cancer, chronic lung disease, or congestive heart failure). We estimated the sensitivity, specificity, and positive predictive value (PPV) of ONYX compared to manual review using multivariable logistic regression, accounting for clustering of reports within people. We examined how ONYX's performance varied by age and comorbidity. Results: ONYX classified 26% of reports (1,276/5,000; 38% [841/2,200] of true pneumonias and 16% [435/2,800] of non-pneumonias) as "requiring manual review" based on pre-defined criteria. Reports from older people were more likely to require manual review than those from younger people. Among reports that could be classified, ONYX had a sensitivity of 91% (1,242/1,359), specificity of 92% (2,170/2,365), and PPV of 81% (1,242/1,437, modeled based on pneumonia prevalence in the source database). Sensitivity and specificity were similar regardless of comorbidity. Conclusions: NLP offers potential for identifying pneumonia outcomes from EMR data. Next steps include 1) further training to decrease the proportion of reports requiring manual review and 2) evaluating the accuracy of ONYX in other health systems.