Gamification has emerged as a promising approach for foreign language learning (FLL), which refers to the use of game design elements to engage learners or improve academic performance. However, the features of gamification studies in FLL and their effectiveness are unclear. Additionally, how previous studies measured the effectiveness of gamified FLL tools is not well understood. In this systematic review, this author addressed these questions based on 21 empirical studies. The findings revealed that the effectiveness of gamified tools in FLL was mixed, with some bringing positive changes, others negative changes, and some showing no differences. The factors that influenced the effectiveness include methodological limitations, biases in the experiment setting, technical limitations, individual differences, failure to achieve meaningful gamification, a mixture of element selection, sub-optimal measurement, and data interpretation biases. This study identified research gaps in previous studies and offers suggestions for future research in this area.