Scientific optical 3D modeling requires the possibility to implement highly flexible and customizable mathematical models as well as high computing power. However, established ray tracing software for optical design and modeling purposes often has limitations in terms of access to underlying mathematical models and the possibility of accelerating the mostly CPU-based computation. To address these limitations, we propose the use of NVIDIA's OptiX Ray Tracing Engine as a highly flexible and high-performing alternative. OptiX offers a highly customizable ray tracing framework with onboard GPU support for parallel computing, as well as access to optimized ray tracing algorithms for accelerated computation. To demonstrate the capabilities of our approach, a realistic focus variation instrument is modeled, describing optical instrument components (light sources, lenses, detector, etc.) as well as the measuring sample surface mathematically or as meshed files. Using this focus variation instrument model, exemplary virtual measurements of arbitrary and standardized sample surfaces are carried out, generating image stacks of more than 100 images and tracing more than 1E9 light rays per image. The performance and accuracy of the simulations are qualitatively evaluated, and virtually generated detector images are compared with images acquired by a respective physical measuring device.