A recently reported vision ray metrology technique [Opt. Express 29, 43480 (2021)OPEXFF1094-408710.1364/OE.443550] measures geometric wavefronts with high precision. This paper introduces a method to convert these wavefront data into height information, focusing on the impact of back surface flatness and telecentricity errors on measurement accuracy. Systematic errors from these factors significantly affect height measurements. Using ray trace simulations, we estimate reconstruction errors with various plano-concave and plano-convex elements. We also developed a calibration technique to mitigate telecentricity errors, achieving submicron accuracy in surface reconstruction. This study provides practical insights into vision ray metrology systems, highlighting validity limits, emphasizing the importance of calibration for larger samples, and establishing system alignment tolerances. The reported technique for the conversion of geometric wavefronts to surface topography employs a direct non-iterative ray-tracing-free method. It is ideally suited for reference-free metrology with application to freeform optics manufacturing.