Due to the falling birth rate and large-scale rural–urban migration, the ageing population in rural China is critical, and the quality of life of the rural elderly needs to be given more attention. In recent years, as an important part of new rural construction in China, the rural environment has been greatly improved, but the impact of such environmental improvement on the health of the rural elderly is not clear. Based on China Health and Retirement Longitudinal Study (CHARLS) data, this paper aims to assess the effect of the improvement of the indoor dwelling environment on cognition among the rural elderly by using a series of the difference-in-difference models (DID) under a counterfactual causality framework. The results showed that first, the improvement of both the toilet type and in-house shower/bath facility had a significant effect on the overall memory and delayed memory among the rural elderly in China, but the effect on immediate memory did not pass the significant test. Second, although access to running water could improve all types of memory, the effect was not significant. In summary, the indoor dwelling environment should be strengthened to ensure the process of healthy ageing in rural China.