Obesity is a potent risk factor for knee osteoarthritis (OA) that is driven by mechanical and potentially endocrine mechanisms, and it affects women more frequently than men. The infrapatellar fat pat (IPFP) represents a potential link between obesity, intra-articular inflammation and structural pathology. Here we investigate whether the IPFP is responsive to body weight loss/gain in women and how its responsiveness to weight change compares to that of subcutaneous fat (SCF) of the thigh. All female participants of the Osteoarthritis Initiative (OAI) with ≥10% weight loss/gain between baseline and a 2-year follow-up were included. Within-subject changes in IPFP volume and SCF cross-sectional areas (CSA) were determined from 3-T magnetic resonance imaging. Linear regression was used to assess the association between change in weight, IPFP volume, and SCF CSA. In the 38 participants with ≥10% weight loss over 2 years (age 59.3 ± 9.1 years, mean loss = 15.9%), there was a significant reduction in IPFP volume (-2.2%, p = 0.02) as well as in SCF CSA (-22%, p < 0.001). In the 34 participants with ≥10% gain (age 61.5 ± 8.7 years, mean gain = 15.9%), there was a significant increase in SCF CSA (+26%, p < 0.001) but not in IPFP volume (0.2%, p = 0.87). Weight change was significantly associated with SCF CSA change (r = 0.76, p < 0.001) but not with IPFP volume change (r = 0.11, p = 0.37). In this first longitudinal, observational study investigating the responsiveness of IPFP and SCF to weight change, IPFP morphology was found responsive to weight loss but not to weight gain. Overall, the responsiveness of the IPFP was substantially less than that of the SCF.