Observational pain scales can help to identify pain in persons with dementia who may have difficulty expressing pain verbally. The Pain Assessment in Impaired Cognition-15 (PAIC15) covers 15 items that indicate pain, but it is unclear how probable pain is, for each summed score (range 0–45). We aimed to determine sensitivity and specificity of cut-offs for probable pain on the PAIC15 against three standards: (1) self-report when able, (2) the established Pain Assessment in Advanced Dementia (PAINAD) cut-off of 2, and (3) observer’s overall estimate based on a series of systematic observations. We used data of 238 nursing home residents with dementia who were observed by their physician in training or nursing staff in the context of an evidence-based medicine (EBM) training study, with re-assessment after 2 months in 137 residents. The area under the ROC curve was excellent against the PAINAD cut-off (≥0.8) but acceptable or less than acceptable for the other two standards. Across standards and criteria for optimal sensitivity and specificity, PAIC15 scores of 3 and higher represent possible pain for screening in practice, with sensitivity and specificity against self-report in the 0.5 to 0.7 range. While sensitivity for screening in practice may be too low, a cut-off of 4 is reasonable to indicate probable pain in research.