The design, implementation, and outcome of game-based learning for learners with disabilities have not been sufficiently examined systematically. Particularly, learner-based and contextual factors, as well as the essential roles played by various stakeholders, have not been addressed when game-based learning applications are used in special education. Therefore, a systematic literature review using the Activity Theory (AT) was conducted to analyse studies about game-based learning for learners with disabilities. Content analysis of 96 studies reported relevant information with respect to each activity component—(a) subject (learners with disabilities), (b) technology (game-based learning applications), (c) object (target skills or behaviours), (d) rules (implementation procedure and performance measures), (e) community (learners with disabilities, special education professionals, and parents), (f) division of labour (among learners, professionals, and parents) and (g) outcome (performance of target skills or behaviours). Furthermore, this study identified existing gaps from the reviewed studies, including occasional lack of parental engagement, difficulty of standardising performance measures due to the heterogeneity of learner profiles and contradictions (e.g., opposing views among experts on the role of educational games in social interactions). Finally, recommendations were made under each activity component. The study concluded that both general and domain-specific guidelines should be created for each disability category proposed in this review to assist practitioners who wish to use game-based learning with learners with disabilities.