Vitiligo is an acquired chronic depigmentation disorder that can have a negative impact on the quality of life (QoL). This is especially true for patients with non-white skin. Only few studies have investigated the QoL of Asian patients with vitiligo. We aimed to investigate the QoL in Taiwanese vitiligo patients and identify the factors that influence their QoL. The cross-sectional study recruited 100 vitiligo patients and 100 controls with general skin diseases in the Department of Dermatology of Changhua Christian Hospital. Data were obtained using a structured questionnaire for demographic information and modified Skindex-21 instruments. The QoL was not significantly different between vitiligo patients and controls. Among the vitiligo patients, adults exhibited deteriorated emotional levels and total QoL as compared with non-adults. Married females reported greater levels of emotional disturbance than the unmarried ones. A higher educational level and shorter history of disease were associated with greater emotional impacts. The patients with a generalized type of vitiligo suffered more in total QoL. After multivariate adjustment, the young adult patients aged 20–39 were associated with poorer total QoL. It is suggested that vitiligo patients who are aged between 20 and 39, are married females, are highly educated, have a shorter disease history, and suffer from the generalized type of this disease demonstrate more deterioration in their life quality compared with other vitiligo patients. Care providers should tailor the psychological counseling and treatment accordingly.