Detecting rare species is important for both threatened species management and invasive species eradication programs. Conservation scent dogs provide an olfactory survey tool that has advantages over traditional visual and auditory survey techniques for some cryptic species. From the literature, we identified 5 measures important in evaluating the use of scent dogs: precision, sensitivity, effort, cost, and comparison with other techniques. We explored the scale at which performance is evaluated and examined when field testing under real working conditions is achievable. We also identified cost differences among studies. We examined 61 studies published in 1976–2018 that reported conservation dog performance, and considered the inconsistencies in the reporting of scent dog performance among these studies. The majority of studies reported some measure of performance; however, only 8 studies reported all 3 aspects necessary for performance evaluation: precision, sensitivity, and effort. Although effort was considered in 43 studies, inconsistent methods and incomplete reporting prevented meaningful evaluation of performance and comparison among studies. Differences in cost between similar studies were influenced by geographical location and how the dog and handler were sourced for the study. To develop consistent reporting for evaluation, we recommend adoption of sensitivity, precision, and effort as standard performance measures. We recommend reporting effort as the total area and total time spent searching and reporting sensitivity and precision as proportions of the sample size. Additionally, reporting of costs, survey objectives, dog training and experience, type of detection task, and human influences will provide better opportunities for comparison within and among studies.