The growing network density and unprecedented increase in network traffic, caused by the massively expanding number of connected devices and online services, require intelligent network operations. Machine Learning (ML) has been applied in this regard in different types of networks and networking technologies to meet the requirements of future communicating devices and services. In this article, we provide a detailed account of current research on the application of ML in communication networks and shed light on future research challenges. Research on the application of ML in communication networks is described in: i) the three layers, i.e., physical, access, and network layers; and ii) novel computing and networking concepts such as Multi-access Edge Computing (MEC), Software Defined Networking (SDN), Network Functions Virtualization (NFV), and a brief overview of ML-based network security. Important future research challenges are identified and presented to help stir further research in key areas in this direction. INDEX TERMS Communication networks, machine learning, physical layer, MAC layer, network layer, SDN, NFV, MEC, security, artificial intelligence (AI) I. INTRODUCTION T HE security, availability and performance demands of new applications, services and devices are increasing at a pace higher than anticipated. Real-time responsiveness in application areas like e-health, traffic, and industry requires communication networks to make real-time decisions autonomously. Such real-time autonomous decision-making requires that the network must react and learn from the environment, and control itself without human interventions. However, communication networks have until now taken a different path. Traditional networks rely on human involvement to respond manually to changes such as traffic variation, updates in network functions and services, security breaches, and faults. Human-machine interactions have resulted in network downtime [1], have opened the network to security vulnerabilities [2], and lead to many other challenges in current communication networks [3], [4]. The requirement for human interaction or manual configuration constitutes a major hindrance for a network to use its past experiences to adapt to changing requirements. The general idea is to predict the (future) behavior of a service, network segment, user or User Equipment (UE), and tune the network at run-time based on this information. For instance, the movement trajectory of a user can be predicted using