Background: Although many genetic and nongenetic factors interact to determine an individual's physical phenotype, there has been limited examination of the contribution of family history of exceptional parental longevity on decline in physical function in aging. Methods: The LonGenity study recruited a relatively genetically homogenous cohort of Ashkenazi Jewish adults age 65 and older, who were defined as either offspring of parents with exceptional longevity ([OPEL]: having at least one parent who lived to age 95 or older) or offspring of parents with usual survival ([OPUS]: neither parent survived to age 95). Decline in performance on objective measures of strength (grip strength), balance (unipedal stance), and mobility (gait speed) as well as a composite physical function measure, the Short physical performance battery (SPPB), were compared between the two groups over a median follow-up of 3.2 years, accounting for age, sex, education, and comorbidities. Results: Of the 984 LonGenity participants (mean age 76, 55% women), 448 were OPEL and 536 were OPUS. Compared to OPUS, OPEL had slower decline on measures of unipedal stance (−0.03 log-units/year, p = .026), repeated chair rise (0.13 s/year, p = .020) and SPPB (−0.11 points/year, p = .002). OPEL women had slower decline on chair rise and SPPB scores compared to OPUS women, although OPEL men had slower decline on unipedal stance compared to OPUS men. Conclusion: Our findings provide evidence that variation in late-life decline in physical function is associated with familial longevity, and may vary for men and women.