Oleogels (defined as structured solid-like materials with a high amount of oil entrapped within a three-dimensional network of gelator molecules) represent a healthy alternative to fats that are rich in saturated and trans fatty acids. Given its fatty acids composition (oleic, linoleic, and linolenic acids), olive oil is an excellent candidate for the use of oleogels in the food industry. In this study, a D-optimal mixture design was employed to optimize the replacement of butter with olive oil-based oleogel in a type of sponge cake formulation: the plum cake. In addition, emulsifiers and whey proteins were used as recipe ingredients to extend the product’s shelf life by delaying staling phenomena and mold growth. In the experimental design, oleogel, emulsifier, and whey protein variables were set as the ingredients that change in specific ranges, while hardness, porosity, water activity, and moistness were used to characterize the obtained formulations. The experimental data of each response were fitted through polynomial regression models with the aim of identifying the best plum cake formulation. The results revealed that the best mixture was the formulation containing 76.98% olive oil-based oleogel, 7.28% emulsifier E471, and 15.73% whey protein. We stored the optimized plum cake for 3 months at room temperature and then checked for any hardness and moistness changes or mold spoilage.