The suspension of face-to-face teaching, due to the COVID-19 social distancing regulations, raised serious concerns about the impacts on children’s academic learning. Because the implementation of distance education in Germany was entirely the responsibility of individual schools, and because the home learning environments varied across households, school children had very different learning conditions during the pandemic. This fact raises questions whether the framework of distance learning has impacted children’s development of basic number skills. In this paper, descriptive information on children’s home learning conditions and teachers’ distance teaching approaches during the pandemic, socio-cultural capital, and basic number skills of 484 third and fourth grade students (51,2% girls) in the state of North Rhine-Westphalia (Germany) are assessed. The data revealed risk factors such as not having a tablet/laptop, lack of access to the internet, or a learning environment with siblings without an adult family member present. A negative association was found between multiple risk factors (at-risk levels) in home learning and basic number skills. This link was partially mediated by socio-cultural capital and moderated by teachers’ distance teaching approach. Children whose teachers applied a more personalized teaching approach showed fewer negative relations between at-risk levels and basic number skills. While no evidence was observed for positive effects of videoconferencing, school-based emergency classes, or private tutoring on basic number skills, children whose learning was supported by learning management systems showed better skills than their peers. The findings highlight the differential impact of home-based learning conditions during the pandemic and provide practical implications for realization of distance teaching.