Several modifications were introduced in a criterion‐referenced evaluation system to determine which modification would result in optimal evaluator reliability and performance differentiation on two representative wax‐up projects. Variables investigated included the use of percentages as cut‐off scores and the number of scoring options. The use of percentage cut‐off scores to define minimal competency significantly increased intrajudge reliability for one of the two projects. Two‐option scoring resulted in significantly higher rater agreement than three‐option scoring in half of the comparisons.