Purpose: Accurate and consistent evaluation of resident operative performance is necessary but difficult to achieve. This study continues examination of the Southern Illinois University (SIU) operative performance rating system (OPRS) for previously unreported factors that may influence reliability, accuracy, and interpretability of results. Methods: OPRS evaluations of surgical residents by faculty at SIU, from 2004 to 2008, were analyzed for the most frequently rated procedures to determine (1) the elapsed time from the procedure until completion of rating, (2) patterns in responses of procedure-specific and global surgical skills items, and (3) whether particular evaluating surgeons differed in their stringency of ratings of resident operative performance. Results: Five-hundred sixty-six evaluations were analyzed: open colectomy (n ϭ 125), open inguinal hernia (n ϭ 103), laparoscopic cholecystectomy (n ϭ 199), and excisional biopsy (n ϭ 139). The number of residents evaluated per training level (PGY) ranged from 88 to 161. Median time to completion of evaluations was 11 days, 9 hours. The quickest evaluation was 18 hours after assignment, but most evaluations were completed within 4.5-22 days. Use of procedure-specific and global scale scores resulted in similar rank ordering of performances (single-measure intraclass correlation using the consistency model ϭ 0.88 [95% confidence interval ϭ 0.87-0.90]) and in similar absolute OPRS scores (0.89 [0.87-0.90]). Evaluating surgeons differed from one another in stringency of ratings across procedures (average difference ϭ 1.4 points on a 5-point scale). Resident ratings increased with PGY level for all 4 procedures.Conclusions: Substantial time elapses between a resident's performance in the operating room and the completion of the evaluation by faculty. This raises the question of whether surgeons remember the nuances of the procedure well enough to rate performance accurately. The item type used for rating operative performances does not affect the absolute rating assigned or the rank ordering of the performance. Differences in stringency of evaluators indicate the need for multiple resident performances to be observed by multiple surgeons. These findings are the foundation for an upcoming multi-institutional trial supported by the American Board of Surgery to examine the effects of elapsed time to evaluation and various resident characteristics when scoring operative performance. Novel hybrid Objective Structured Assessing Technique (OSAT)/ Objective Structured Clinical Examinations (OSCE) in comprehensive perioperative breast care: a three-year analysis of outcomes.Maker VK, Bonne S.