Energy saving and visual comfort are two main considerations in designing of automatic illumination control systems. However, energy-saving-oriented illumination control always causes optical spectra drifting in light-conversion-material-based white light-emitting diodes (WLEDs), which are conventionally used as artificial luminaires in indoor areas. In this study, we propose a method for InP quantum dot (QD)-based WLEDs to minimize optical energy consumption by considering the influence caused by the outdoor environment and neighboring WLED units. Factors of (a) dimensions of room window and WLED matrix, (b) distance between WLED units, lighting height, species of InP QDs, and (c) user distribution are taken into consideration in calculation. Parameters of correlated color temperature (CCT) and color rendering index (Ra) of the WLED matrix are optimized according to the lighting environment to improve user visual comfort level. By dynamically controlling the light ingredients and optical power of WLEDs, we optimize the received illuminance distribution of table tops, improve the lighting homogeneity of all users, and guarantee the lowest energy consumption of the WLED matrix. The proposed approach can be flexibly applied in large-scale WLED intelligent controlling systems for industrial workshops and office buildings.