Objective: The relationships between physical activity (PA) and physical function (PF) among older Japanese adults have been examined before, with some studies reporting that moderate-to-vigorous physical activity (MVPA) is positively associated with PF. However, it is unclear whether the observed associations differ between men and women. In this study, we investigated the associations of objectively measured MVPA, light physical activity (LPA), and sedentary time (ST) with different PF levels in older Japanese men and women. Subjects and Methods: A total of 810 community-dwelling subjects aged 65–75 years were investigated (52.1% women, 47.9% men). The PF variables included grip strength, one-leg standing, usual and maximum walking speeds, and chair-standing time. PA (MVPA, LPA, and ST) and the number of steps taken daily were assessed for seven consecutive days by a tri-axial accelerometer. We determined the sex-specific quartiles of MVPA, LPA, and ST and analyzed their associations with physical function in separate models for each exposure measure. Results: In the crude analysis, MVPA was significantly associated with all PF variables in the men, and with the usual walking speed, max. walking speed, and chair-standing time in the women. Neither LPA nor ST was significantly associated with any PF variables. After adjusting potential confounding factors, significant associations between MVPA and usual walking speed remained in the men and women. Conclusions: Only greater moderate-to-vigorous physical activity (MVPA) was significantly associated with higher levels of PF variables in both men and women. Thus, time spent in moderate-to-vigorous physical activity (MVPA) can help older adults maintain or improve their physical function.