Retail platforms have widely implemented recommender systems to provide personalized recommendations to consumers, influencing sales significantly. However, under the hybrid selling mode where platforms offer both their products and third-party sellers’ products, the profitability of a recommender system and the optimal allocation of recommendations become critical considerations. This paper introduces a game-theoretic model to investigate these issues and unveil how a recommender system and its characteristics influence prices and profits. A key finding is that the recommender system increases prices and profits only if the commission rate is high and the system is profit-oriented or inaccurate. Surprisingly, higher recommendation accuracy does not always translate into higher profits; it is advantageous only in a consumer-oriented system. Moreover, the retail platform tends to allocate more recommendations to its own product than to the third-party seller’s product, a strategy known as self-preferencing. This strategy gives the platform a competitive edge and boosts its profit compared to the third-party seller. Furthermore, the degree of self-preferencing varies with the accuracy and orientation of the recommendation system. Specifically, in a consumer-oriented system, self-preferencing increases with accuracy, while in a profit-oriented system, it decreases with accuracy.