Advances in artificial intelligence (AI) have shaped today’s user services, enabling enhanced personalization and better support. As such AI-based services inevitably require user data, the resulting privacy implications are de facto the unacceptable face of this technology. In this article, we categorize and survey the cutting-edge research on privacy and data protection in the context of personalized AI services. We further review the different protection approaches at three different levels, namely, the management, system, and AI levels—showing that (i) not all of them meet our identified requirements of evolving AI services and that (ii) many challenges are addressed separately or fragmentarily by different research communities. Finally, we highlight open research challenges and future directions in data protection research, especially that comprehensive protection requires more interdisciplinary research and a combination of approaches at different levels.