BackgroundThere are various definitions and diagnostic criteria for dementia, leading to discrepancies in case ascertainment in both clinical practice and research. We reviewed the different definitions, approaches and measurements used to operationalize dementia in health care studies in German nursing homes with the aim of discussing the implications of different approaches.MethodsWe conducted a systematic search of the MEDLINE and CINAHL databases to identify pre-2016 studies conducted in German nursing homes that focused on residents with dementia or cognitive impairment. In- or exclusion of studies were consented by all authors; data extraction was independently carried out by 2 authors (RP, SJ). The studies’ sampling methods were compared with respect to their inclusion criteria, assessment tools and methods used to identify the study population.ResultsWe summarized case ascertainment methods from 64 studies. Study participants were identified based on a diagnosis that was evaluated during the study, or a recorded medical dementia diagnosis, or a recorded medical diagnosis either with additional cognitive screenings or using screening tests exclusively. The descriptions of the diagnostics that were applied to assess a diagnosis of dementia were not fully transparent in most of the studies with respect to either a clear reference definition of dementia or applied diagnostic criteria. If reported, various neuropsychological tests were used, mostly without a clear rationale for their selection.ConclusionPragmatic considerations often determine the sampling strategy; they also may explain the variances we detected in the different studies. Variations in sampling methods impede the comparability of study results. There is a need to consent case ascertainment strategies in dementia studies in health service research in nursing homes. These strategies should consider resource constraints and ethical issues that are related to the vulnerable population of nursing home residents. Additionally, reporting about dementia studies in nursing homes need to be improved. If a diagnosis cannot be evaluated based on either ICD or DSM criteria, the study population may not be reported as having dementia. If a diagnosis is evaluated based on ICD or DSM criteria within the study, there is a need for more transparency of the diagnostic process.Electronic supplementary materialThe online version of this article (doi:10.1186/s12877-016-0249-7) contains supplementary material, which is available to authorized users.