One of the promising technologies to support the application of mathematics learning is augmented reality (AR). It is considered an important pedagogical tool that allows an increasing understanding of challenging ideas at most levels of education. This article presents the approach and concept of a systematic literature review (SLR) for reviewing the effects of AR in mathematics education. Filtering relevant material on AR and mathematics education from two databases (Scopus and Eric) to answer research questions is part of the review study. In the investigation, a total of 23 publications from 2018 to 2022 were systematically selected based on the PRISMA protocol. A review of the literature shows that interest in AR research has grown over time and is evenly distributed across different countries. The use of AR in mathematics education has been adopted and used as a supporting medium for interactive learning at various levels, from elementary school to college, that appears on the topics of geometry, algebra, basic mathematics, statistics and probability, and other mathematical topics. The effectiveness of AR, which is widely developed by researchers, is its ability to overcome existing problems, such as learning barriers, mathematical anxiety, and other cognitive problems. This review has filled and amplified the literature on AR on the effectiveness of AR in school mathematics learning. We recommend that in the future research on AR should focus on exploring the broad uses and long-term impacts of AR development and implementation on mathematics learning.