The Industrial Revolution has shifted towards Industry 5.0, reinventing the Industry 4.0 operational process by introducing human elements into critical decision processes. Industry 5.0 would present massive customization via transformative technologies like cyber-physical systems, artificial intelligence (AI), and big-data analytics. In Industry 5.0, the AI models must be transparent, valid, and interpretable. AI models employ machine learning (ML) and deep learning (DL) mechanisms to make the industrial process autonomous, reduce downtime, and improve operational and maintenance costs. However, the models require explainability in the learning process. Thus explainable AI (EXAI) adds interpretability and improves the diagnosis of critical industrial processes, which augments the machine-to-human (M2H) explanations and vice-versa. Recent surveys of EXAI in industrial applications are mostly oriented towards EXAI models, the underlying assumptions. Still, fewer studies are conducted towards a holistic integration of EXAI with human-centric processes that drives the Industry 5.0 applicative verticals. Thus, to address the gap, we propose a first-of-its-kind survey that systematically untangles EXAI integration and its potential in Industry 5.0 applications. First, we present the background of EXAI in Industry 5.0 and cyber-physical systems (CPS) and a reference EXAI-based Industry 5.0 architecture with insights into Large Language Models (LLMs). Then, based on the research questions, a solution taxonomy of EXAI in Industry 5.0 is presented, which is ably supported by applicative use cases (cloud, digital twins, smart grids, augmented reality, and unmanned aerial vehicles). Finally, a case study of EXAI in manufacturing cost assessment is discussed, followed by open issues and future directions. The survey is designed to extend novel prototypes and designs to realize EXAI-based in real-time Industry 5.0 applications.
