Massive progress of mobile wireless telecommunication networks was achieved in the previous decades, with privacy enhancement in each. At present, mobile users are getting familiar with the latest 5G networks, and the discussion for the next generation of Beyond 5G (B5G)/6G networks has already been initiated. It is expected that B5G/6G will push the existing network capabilities to the next level, with higher speeds, enhanced reliability, and seamless connectivity. To make these expectations a reality, research is progressing on new technologies, architectures, and intelligence-based decision-making processes related to B5G/6G. Privacy considerations are a crucial aspect that needs further attention in such developments, as billions of people and devices will be transmitting their data through the upcoming network. This paper provides a comprehensive survey on privacy-related aspects for B5G/6G networks. First, it discusses a taxonomy of different privacy perspectives. Based on the taxonomy, the paper then conceptualizes a set of privacy goals for the B5G/6G and the challenges that appear as barriers to reaching these goals. Next, this work provides a set of solutions applicable to the proposed architecture of B5G/6G networks to mitigate the challenges. Additionally, this paper discusses the emerging field of non-personal data privacy. It also provides an overview of standardization initiatives for privacy preservation. Finally, it concludes with a roadmap of future directions and upcoming trends containing privacy-related topics, which will be an arena for new research towards privacy-enhanced B5G/6G networks. This work provides a basis for privacy aspects that will significantly impact peoples' daily lives with future networks.