Adipose tissue is an organ with metabolic, endocrine and immune functions. In this tissue, the expressions of genes associated with several metabolic pathways, including lipid metabolism, have been shown to be affected by genetic selection for feed efficiency, an important trait to consider in livestock. We hypothesized that the stimulation of immune system caused by poor hygiene conditions of housing impacts the molecular and cellular features of adipose tissue and that the impact may differ between pigs that diverge in feed efficiency. At the age of 12 weeks, Large White pigs from two genetic lines divergent for residual feed intake (RFI) were housed in two contrasting hygiene conditions (good vs poor). After six weeks of exposure, pigs were slaughtered (n = 36). Samples of blood, subcutaneous (SCAT) and perirenal (PRAT) adipose tissues were collected for cell response and gene expression investigations. The decrease in the relative weight of PRAT was associated with a decline in mRNA levels of FASN, ME, LCN2 and TLR4 (P < 0.05) in pigs housed in poor conditions compared with pigs housed in good conditions for both RFI lines. In SCAT, the expressions of only two key genes (PPARG and TLR4) were significantly affected by the hygiene of housing conditions. Besides, the mRNA levels of both LCN2 and GPX3 were influenced by the RFI line (P < 0.05). Because we suspected an effect of poor hygiene at the cellular levels, we investigated the differentiation of stromal vascular cells isolated from SCAT in vitro in the absence or presence of a pro-inflammatory cytokine, Tumor Necrosis Factor-α (TNF-α). The ability of these cells to differentiate in the absence or presence of TNF-α did not differ among the four groups of animals (P > 0.05). We also investigated the expressions of genes involved in the immune response and lipid metabolism in whole blood cells cultured in the absence and presence of LPS. The hygiene conditions had no effect but, the relative expression of the GPX3 gene was higher (P < 0.001) in high RFI than in low RFI pigs while the expressions of IL-10 (P = 0.027), TGFβ1 (P = 0.023) and ADIPOR2 (P = 0.05) genes were lower in high RFI than in low RFI pigs. Overall, the current study indicates that the hygiene of housing had similar effects on both RFI lines on the expression of genes in adipose tissues and on the features of SCAT adipose cells and whole blood cells in response to TNF-α and LPS. It further demonstrates that the number of genes with expression impacted by housing conditions was higher in PRAT than in SCAT. It suggests a depot-specific response of adipose tissue to the current challenge.