The digital transformation has profoundly affected every facet of human life, with technological advancements potentially reshaping the economy, society, and our daily living and working modalities. Artificial Intelligence (AI), particularly Generative AI (GAI), has emerged as a pivotal disruption in education, showcasing the capability to produce diverse and context-relevant content. Generative Artificial Intelligence (GAI) has revolutionized natural language processing, computer vision, and creative arts. Large language models (LLMs) like GPT-4 and Open Assistant and tools like DALL-E and Midjourney for the visual and creative domain are increasingly used for various tasks by students and others with critical information needs. AI presents novel avenues for crafting effective learning activities and developing enhanced technology-driven learning applications in the educational sector. However, integrating AI with a pedagogical focus pose challenge. Education 4.0, which integrates emerging technologies and innovative strategies, aims to prepare new generations for a technologically fluid world. This systematic literature review aims to analyze the use of LLMs in higher education within the context of Education 4.0’s pedagogical approaches, identifying trends and challenges from a selection of 83 relevant articles out of an initial set of 841 papers. The findings underscore the significant potential of LLMs to enrich higher education, aligning with Education 4.0 by fostering more autonomous, collaborative, and interactive learning. It highlights the necessity for human oversight to ensure the quality and accuracy of AI-generated content. It addresses ethical and legal challenges to ensure equitable implementation, suggesting an exploration of LLM integration that complements human interaction while maintaining academic integrity and pedagogical foundation.