Vehicular ad hoc networks (VANETs) have emerged as a powerful network model for providing road safety, as well as infotainment applications, for vehicle passengers and drivers alike. These networks are characterized by high node mobility, which in turn introduces communication intermittency and unreliability, deteriorating the network's performance. To provide solutions to the concerns of performance of such networks, articles have proposed intelligent approaches to deal with such issues by means of artificial intelligence and machine learning techniques. This study aimed to review the literature regarding intelligent routing protocols in VANETs by focusing on the techniques used and on how the solutions were evaluated based on performance. A systematic literature review of studies was conducted by adopting the snowballing procedure to collect studies that propose novel approaches to solving the VANET routing protocol by means of intelligent algorithms. The 86 included studies reported that heuristics, fuzzy logic systems and reinforcement learning approaches are the most popular and effective methods to improve network performance in the dynamic VANET environment. The findings have also shown that the literature has yet to find a consensus as to how to evaluate routing protocol performance. An evaluation and comparison framework is required as to enable transparent routing protocol design and selection in future vehicle applications.