Having a healthy built environment becomes increasingly important, especially under the effects of COVID-19. This paper intends to combine sustainable goals based on climate change with passive design principles to achieve a healthy built environment regarding the building performance of residential buildings. The Yuedao Residential Community in the Lingnan area was taken as an example for the research. Based on relevant standards of healthy buildings, the thermal, light, and acoustic environment requirements were determined. The methods of building performance simulation and on-site measurement were used to quantify the research object environments. Then, the outcomes were obtained based on these standards. As observed, the thermal environment’s adaptive thermal comfort level was level III. It was hot indoors, but the light and acoustic environments met the requirements. Building designs based on a built environment optimized by external shading systems aim to solve problems through building performance simulation and qualitative analysis. After optimization, the thermal environment improved. According to the literature review, this research focused on a healthy built environment with a sustainable passive design in terms of building performance. A research workflow was established that could be used for more practical research, with abundant research methods. The problems were solved to varying degrees, and the Lingnan architectural culture was preserved. Moreover, this research filled the gap in interactive research on healthy built environments with sustainable passive design regarding building performance.