The principle of education for sustainable development (ESD) is that no child is left behind. Hence, the fourth sustainable development goal (SDG) of the United Nations (UN) emphasizes inclusion and equity in education by focusing on eliminating disparities among regions. This study explores factors related to education quality through modeling in rural areas of Sumenep Regency, in East Java, Indonesia. Currently, only a few kinds of research studies involve spatial data, latent variables and, at the same time, tests of their spillover effects. The modeling herein is the spatial autoregressive model with latent variables (SAR-LVs). The latent variables were estimated using the weighted least square (WLS) method, while the Lagrange multiplier (LM) test was used for spatial dependence testing. The parameters of the SAR-LVs were estimated using two-stage least square (2SLS). The results show that the quality of education is directly influenced by the infrastructure of the schools but not by the socioeconomic conditions of the local communities. The autoregressive spatial coefficient has a significant but negative effect, which shows a negative spillover from districts with a lower quality of education to the ones with a high quality of education. This is due to the students’ competition to get registered for a favorite or good quality school in a particular district, which stimulates the migration of students from its neighboring districts. This reveals the inequality of school quality, since not all students can get access to schools with good quality. Through this study, some recommendations are given as a contribution to achieving the fourth SDG in Indonesia.