The IETF Routing Over Low power and Lossy network (ROLL) working group defined IPv6 Routing Protocol for Low Power and Lossy Network (RPL) to facilitate efficient routing in IPv6 over Low-Power Wireless Personal Area Networks (6LoWPAN). Limited resources of 6LoWPAN nodes make it challenging to secure the environment, leaving it vulnerable to threats and security attacks. Machine Learning (ML) and Deep Learning (DL) approaches have shown promise as effective and efficient mechanisms for detecting anomalous behaviors in RPL-based 6LoWPAN. Therefore, this paper systematically reviews and critically analyzes the research landscape on ML, DL, and combined ML-DL approaches applied to detect attacks in RPL networks. In addition, this study examined existing datasets designed explicitly for the RPL network. This work collects relevant studies from five major databases: Google Scholar, Springer Link, Scopus, Science Direct, and IEEE Xplore® digital library. Furthermore, 15,543 studies, retrieved from January 2016 to mid-2021, were refined according to the assigned inclusion criteria and designed research questions resulting in 49 studies. Finally, a conclusive discussion highlights the issues and challenges in the existing studies and proposes several future research directions.