The aim of this study was to assess the characteristics of visual search behavior in elderly drivers in reverse parking. Fourteen healthy elderly and fourteen expert drivers performed a perpendicular parking task. The parking process was divided into three consecutive phases (Forward, Reverse, and Straighten the wheel) and the visual search behavior was monitored using an eye tracker (Tobii Pro Glasses 2). In addition, driving-related tests and quality of life were evaluated in elderly drivers. As a result, elderly drivers had a shorter time of gaze at the vertex of the parking space both in direct vision and reflected in the driver-side mirror during the Forward and the Reverse phases. In contrast, they had increased gaze time in the passenger-side mirror in the Straighten the wheel phase. Multiple regression analysis revealed that quality of life could be predicted by the total gaze time in the Straighten the wheel phase (β = −0.45), driving attitude (β = 0.62), and driving performance (β = 0.58); the adjusted R2 value was 0.87. These observations could improve our understanding of the characteristics of visual search behavior in parking performance and how this behavior is related to quality of life in elderly drivers.