Banana and bergamot peels are underutilized byproducts of the essential oil and juice-processing industry. This study was designed for the development of rigid polyurethane foam (RPUF) composites using polysaccharide-rich fruit peels as fillers. These fillers were characterized for chemical properties using wet analyses. Additionally, the influences of the filler type and filler content on morphological, thermal, mechanical, hygroscopic, and colorimetric properties of the RPUF were investigated. The main results indicated that, in a comparison with the neat RPUF, the insertion of up to 15% of fillers yielded similar water uptake, apparent density, compressive strength, and color properties, as well as increases up to 115% in thermal stability and up to 80% in cell size.