The infrapatellar fat pad (IFP) is actively involved in knee osteoarthritis (OA). However, a proper description of which developmental modifications occur in the IFP along with age and in absence of joint pathological conditions, is required to adequately describe its actual contribution in OA pathophysiology. Here, two IFP sources were compared: (a) IFP from healthy young patients undergoing anterior-cruciate ligament (ACL) reconstruction for ACL rupture (n = 24); (b) IFP from elderly cadaver donors (n = 23). After histopathological score assignment to confirm the absence of inflammatory features (i.e., inflammatory infiltrate and increased vascularity), the adipocytes morphology was determined; moreover, extracellular matrix proteins were studied through histology and Second Harmonic Generation approach, to determine collagens content and orientation by Fast Fourier Transform and OrientationJ. The two groups were matched for body mass index. No inflammatory signs were observed, while higher area, perimeter, and equivalent diameter and volume were detected for the adipocytes in the elderly group. Collagen III displayed higher values in the young group and a lower total collagen deposition with aging was identified. However, collagen I/III ratio and the global architecture of the samples were not affected. A higher content in elastic fibers was observed around the adipocytes for the ACL-IFPs and in the septa cadaver donor-IFPs, respectively. Age affects the characteristics of the IFP tissue also in absence of a pathological condition. Variable mechanical stimulation, depending on age-related different mobility, could be speculated to exert a role in tissue remodeling.