Background: Children and adolescents with intellectual disabilities (ID) have low levels of physical activity (PA). Understanding factors influencing the PA participation of this population is essential to the design of effective interventions. The purposes of this study were to identify and map the barriers and facilitators of PA participation among children and adolescents with ID. Methods: A scoping review was conducted in accordance with established methodology. Articles were evaluated for relevance using predetermined inclusion criteria in eight databases. Extracted barriers and facilitators were classified using the social ecological model as individual, interpersonal, or environmental factors. Results: Thirty-two studies published between 1992 and 2020 were included (24 quantitative, 6 qualitative, and 2 mixed-method). Thirty-four factors were identified. The most commonly reported barriers included disability-specific factors, low self-efficacy, lack of parental support, inadequate or inaccessible facilities, and lack of appropriate programs. The most commonly reported facilitators included high self-efficacy, enjoyment of PA, sufficient parental support, social interaction with peers, attending school physical education (PE) classes, and adapted PA programs. Conclusions: Continued exploration of factors influencing PA participation is required among children and adolescents with ID. Future interventions should involve families, schools, and wider support network in promoting their PA participation together.