Objective To identify which clinical features and personality traits are associated with quality of life (QoL) in panic disorder (PD) patients. Methods This was a cross-sectional study with PD patients. The brief version of the World Health Organization Quality of Life Questionnaire (WHOQOL-BREF) and the Big Five Inventory (BFI) were used to assess QoL and personality traits respectively. The strength of correlations was measured with Pearson’s, Spearman’s, and point-biserial correlation coefficients. We also performed multiple linear regressions, considering sociodemographic data and scores from clinical scales as independent variables and QoL scores as dependent variables. Results A total of 98 patients were evaluated. Depressive symptoms had a strong negative correlation with QoL and, to a lesser extent, panic and anxiety symptoms were also negatively correlated with QoL. While consciousness, extraversion, and agreeableness had mild positive correlations with QoL, neuroticism had a strong negative correlation. Conclusion Symptoms of depression, anxiety, and panic seem to have a negative impact on the QoL of PD patients. Personality traits, especially neuroticism, may also influence QoL in these patients.