Background The value of using comprehensive but cumbersome coding instruments to assess therapeutic competency is unclear. Shorter, more general instruments may enable more research in this important area. The aim of this study was therefore to psychometrically evaluate a shorter version of the Cognitive Therapy Scale-Revised (CTSR) and to compare it with the full-length version. Methods A four-item coding instrument (the CTSR-4) was derived from the CTSR. Four experienced psychotherapists used the CTSR-4 to assess 50 fifteen-minutes samples from audio-recorded CBT sessions. The criterion validity of the CTSR-4 was analyzed by comparing the results with previously expert-rated CTSR scores from the same sessions, and the inter-rater agreement between the three coders was calculated. Results The CTSR-4 showed good criterion validity (ICC = .71–.88) when compared to the expert ratings of the complete CTSR, and the inter-rater agreement was adequate (ICC = .64–.79). Conclusions A condensed version of the CTSR, used to assess CBT competence from shorter samples of therapy sessions, is moderately reliable and may provide similar results as the full-length version. According to preliminary analyses, the CTSR-4 has potential as a low-cost alternative to assess CBT competency in both research and psychotherapist training.