Sweet buttermilk, a by-product of butter production, remains highly underutilized despite containing some relevant components (i.e., phospholipids) that may have a high biological value and may exert some positive technological functions. The aim of this study was to investigate the possibility of using ultrafiltered (UF) sweet buttermilk at different volume reduction ratios (3 and 5) to replace emulsifiers and/or stabilizers in the production of a novel clean-label ice cream formulation made with sweet buttermilk-based mixtures. The functional, thermo-rheological, and sensory profile of four types of ice creams was investigated. Increasing the degree of sweet buttermilk concentration positively influenced the overrun values and at the same time improved the ice cream’s resistance to melting. Also, the thermo-rheological profile during melting was influenced by the presence of UF buttermilk. These differences in techno-functional properties were probably partly caused by the different total phospholipids content caused by UF buttermilk. Some sensory properties (i.e., structure, consistency) were positively related to the utilization of UF buttermilk, while aroma and taste were negatively influenced. This study demonstrated that UF buttermilk can be used as an additive replacer in ice cream production because it enhances the structural and rheological properties of the final product.