The digital divide between rural and urban areas is becoming the key factors resulting educational imbalance, which might be exacerbated by differences in teachers’ digital teaching competence. Therefore, it was crucial to explore the divide and determinants of digital teaching competence between rural and urban teachers. A large-scale survey was conducted with 11,784 K–12 teachers in China (43.40% from rural schools and 56.60% from urban schools). First, this study investigated potential factors for teachers’ digital teaching competence, including information and communication technology (ICT) attitude, ICT skills, and data literacy. Second, the data indicated the digital divide existed, i.e., the ICT attitude, ICT skills, data literacy, and digital teaching competence of rural teachers were significantly lower than those of urban teachers. Third, the Blinder-Oaxaca decomposition method demonstrated that data literacy and ICT skills were the most important determinants of the divide in digital teaching competence between rural and urban teachers. Hence, our research provided important insights for policymakers, school leaders and teachers to bridge the digital divide.