Familial pulmonary fibrosis (FPF) is a monogenic disease most commonly involving telomere- (TERT) or surfactant- (SFTP) related mutations. These mutations have been shown to alter lymphocytic inflammatory responses, and FPF biopsies with histological lymphocytic infiltrates have been reported. Recently, a model of a surfactant mutation in mice showed that the disease initially started with an inflammatory response followed by fibrogenesis. Since inflammation and fibrogenesis are targeted by different drugs, we investigated whether the degree of these two features co-localize or occur independently in different entities of FPF, and whether they influence survival. We quantified the number of lymphocyte aggregates per surface area, the extent of diffuse lymphocyte cell infiltrate, the number of fibroblast foci per surface area, and the percentage of fibrotic lung surface area in digitally scanned hematoxylin and eosin (H&E) sections of diagnostic surgical biopsies of patients with TERT-related FPF (TERT-PF; n = 17), SFTP-related FPF (SFTP-PF; n = 7), and sporadic idiopathic pulmonary fibrosis (sIPF; n = 10). For comparison, we included biopsies of patients with cellular non-specific interstitial pneumonia (cNSIP; n = 10), an inflammatory interstitial lung disease with high lymphocyte influx and usually responsive to immunosuppressive therapy. The degree of inflammatory cell infiltrate and fibrosis in TERT-PF and SFTP-PF was not significantly different from that in sIPF. In comparison with cNSIP, the extent of lymphocyte infiltrates was significantly lower in sIPF and TERT-PF, but not in SFTP-PF. However, in contrast with cNSIP, in sIPF, TERT-PF, and SFTP-PF, diffuse lymphocyte cell infiltrates were predominantly present and lymphocyte aggregates were only present in fibrotic areas (p < 0.0001). Furthermore, fibroblast foci and percentage of fibrotic lung surface were associated with survival (p = 0.022 and p = 0.018, respectively), while this association was not observed for lymphocyte aggregates or diffuse lymphocytic infiltration. Inflammatory cells in diagnostic lung biopsies of TERT-PF, SFTP-PF, and sIPF were largely confined to fibrotic areas. However, based on inflammation and fibrosis, no differences were found between FPF and sIPF, substantiating the histological similarities between monogenic familial and sporadic disease. Furthermore, the degree of fibrosis, rather than inflammation, correlates with survival, supporting that fibrogenesis is the key feature for therapeutic targeting of FPF.