<b><i>Introduction:</i></b> Directional deep brain stimulation (DBS) leads have become widely used in the past decade. Understanding the asymmetric stimulation provided by directional leads requires precise knowledge of the exact orientation of the lead in respect to its anatomical target. Recently, the DiODe algorithm was developed to automatically determine the orientation angle of leads from the artifact on postoperative computed tomography (CT) images. However, manual DiODe results are user-dependent. This study analyzed the extent of lead rotation as well as the user agreement of DiODe calculations across the two most common DBS systems, namely, Boston Scientific’s <i>Vercise</i> and Abbott’s <i>Infinity</i>, and two independent medical institutions. <b><i>Methods:</i></b> Data from 104 patients who underwent an anterior-facing unilateral/bilateral directional DBS implantation at either Northwestern Memorial Hospital (NMH) or Albany Medical Center (AMC) were retrospectively analyzed. Actual orientations of the implanted leads were independently calculated by three individual users using the DiODe algorithm in Lead-DBS and patients’ postoperative CT images. The deviation from the intended orientation and user agreement were assessed. <b><i>Results:</i></b> All leads significantly deviated from the intended 0° orientation (<i>p</i> < 0.001), regardless of DBS lead design (<i>p</i> < 0.05) or institution (<i>p</i> < 0.05). However, the Boston Scientific leads showed an implantation bias toward the left at both institutions (<i>p</i> = 0.014 at NMH, <i>p</i> = 0.029 at AMC). A difference of 10° between at least two users occurred in 28% (NMH) and 39% (AMC) of all Boston Scientific and 76% (NMH) and 53% (AMC) of all Abbott leads. <b><i>Conclusion:</i></b> Our results show that there is a significant lead rotation from the intended surgical orientation across both DBS systems and both medical institutions; however, a bias toward a single direction was only seen in the Boston Scientific leads. Additionally, these results raise questions into the user error that occurs when manually refining the orientation angles calculated with DiODe.