Background: Globally, prostate cancer is the second leading cause of cancer deaths among males. It is the most commonly diagnosed cancer in Australia. The quality of life of prostate cancer patients is poorer when compared to the general population due to the disease itself and its related complications. However, there is limited research on the geographic pattern of quality of life and its risk factors in Victoria. Therefore, an examination of the spatio-temporal pattern and risk factors of poor quality of life, along with the impact of spatial weight matrices on estimates and model performance, was conducted. Method: A retrospective study was undertaken based on the Prostate Cancer Outcome Registry—Victoria data. Patient data (n = 5238) were extracted from the Prostate Cancer Outcome Registry, a population-based clinical quality outcome assessment from 2015 to 2021. A Bayesian spatio-temporal multilevel model was fitted to identify risk factors for poor quality of life. This study also evaluated the impact of distance- and adjacency-based spatial weight matrices. Model convergence was assessed using Gelman–Rubin statistical plots, and model comparison was based on the Watanabe–Akaike Information Criterion. Results: A total of 1906 (36.38%) prostate cancer patients who had undergone surgery experienced poor quality of life in our study. Belonging to the age group between 76 and 85 years (adjusted odds ratio (AOR) = 2.90, 95% credible interval (CrI): 1.39, 2.08), having a prostate-specific antigen level between 10.1 and 20.0 (AOR = 1.33, 95% CrI: 1.12, 1.58), and being treated in a public hospital (AOR = 1.35, 95% CrI: 1.17, 1.53) were significantly associated with higher odds of poor quality of life. Conversely, residing in highly accessible areas (AOR = 0.60, 95% CrI: 0.38, 0.94) was significantly associated with lower odds of poor prostate-specific antigen levels. Variations in estimates and model performance were observed depending on the choice of spatial weight matrices. Conclusion: Belonging to an older age group, having a high prostate-specific antigen level, receiving treatment in public hospitals, and remoteness were statistically significant factors linked to poor quality of life. Substantial spatio-temporal variations in poor quality of life were observed in Victoria across local government areas. The distance-based weight matrix performed better than the adjacency-based matrix. This research finding highlights the need to reduce geographical disparities in quality of life. The statistical methods developed in this study may also be useful to apply to other population-based clinical registry settings.