Milk and dairy ingredients are prime sources for frozen dessert (e.g., ice cream) formulation. However, cow's milk allergy, lactose intolerance, and vegetarianism restrict people from consuming such foods. This review aims to provide an overview of the dairy-free alternatives used in ice cream preparation as well as issues concerning their use in such products with potential solutions. Plant-based ingredients such as plant-based milk, plant oils, and plant proteins are being extensively used in ice cream, and the literature has shown the improved physicochemical properties and additional health benefits of these dairy-free products compared to their dairy counterparts. However, certain issues, such as low nutritive quality, the presence of antinutritional factors, and a lower sensory perception, affect consumer acceptance of these foods. From a product manufacturing viewpoint, future efforts should be made to understand the sensory and nonsensory aspects of these dairy-free products to enable consumers to select products according to their needs.