Cell phone call operations during driving can lead to distraction and cause potential safety hazards. In order to quantitatively characterize the influence of cell phone call mode on the driver’s visual behavior in different traffic conditions and to analyze the risk level of distracted driving from the visual level, a distracted driving simulation test was carried out based on a driving simulator and eye tracker. The eye movement data of drivers during normal driving, hands-free call, and video call under two typical traffic conditions of free flow and congested flow on the urban expressway were collected. Firstly, four visual characteristics indicators that were highly sensitive to traffic conditions and driving states were selected in terms of visual field range, visual recognition, visual search, and visual load, which were the information entropy of fixation area (IEFA), saccade amplitude, peak-to-average ratio of saccade velocity (PARSV), and relative change intensity of pupil area (RCPA). Then, based on the improved CRITIC method, the visual stability coefficient (VSC) was constructed as a new indicator to comprehensively assess the risk level of the driving state, and the assessment criteria were divided. Finally, the grey correlation analysis method was introduced to verify the assessment effect of VSC. The results show that different cell phone call modes increased driving risk in both traffic conditions. Among them, the negative influence of video calls on driving safety was significantly higher than that of hands-free calls, with a significant decrease in VSC, and the drivers’ VSC in the free flow scenario was more sensitive to the impact of cell phone call operation, and the driving risk increased significantly during distracted driving. The VSC can quantitatively assess driving risk from the perspective of visual psychological safety and contributes to the development of corresponding early warning and control measures.