This study aimed to determine the change in saccadic eye movement (SEM) speed according to age (young older; 65–72 years, middle older; 73–80 years, old older: over 81 years) in the elderly and identify the correlation between SEM speed and balance ability. We recruited 128 elderly individuals and measured their SEM speed and balance. The SEM speed was measured to allow the target to appear once every 2 s (0.5 Hz), twice per second (2 Hz), or thrice per second (3 Hz). The SEM performance time was 1 min with a washout period of 1 min. Balance ability was measured using the functional reach test (FRT), timed up-and-go test (TUG), and walking speed (WS). As age increased, FRT, TUG, and WS decreased and SEM speed was significantly decreased in old older than in young older adults at 3 HZ. In all participants, the 3 Hz SEM speed was significantly correlated with TUG and WS. Therefore, SEM speed may be inadequate or decreased in response to rapid external environmental stimuli and may be a factor that deteriorates the ability to balance in older adults.