Around 2020, 5G began its commercialization journey, and discussions about the nextgeneration networks (such as 6G) emerged. Researchers predict that 6G networks will have higher bandwidth, coverage, reliability, energy efficiency, and lower latency, and will be an integrated "humancentric" network system powered by artificial intelligence (AI). This 6G network will lead to many real-time automated decisions, ranging from network resource allocation to collision avoidance for self-driving cars. However, there is a risk of losing control over decision-making due to the high-speed, data-intensive AI decision-making that may go beyond designers' and users' comprehension. To mitigate this risk, explainable AI (XAI) methods can be used to enhance the transparency of the black-box AI decisionmaking process. This paper surveys the application of XAI towards the upcoming 6G age, including 6G technologies (such as intelligent radio and zero-touch network management) and 6G use cases (such as industry 5.0). Additionally, the paper summarizes the lessons learned from recent attempts and outlines important research challenges in applying XAI for 6G use cases soon.