Background Diabetes is one of the biggest worldwide health emergencies of the 21st century. A major goal in the management of diabetes is to prevent diabetic complications that occur as a result of poor glycemic control. Identification of factors contributing to poor glycemic control is key to institute suitable interventions for glycemic control and prevention of chronic complications. Methods A hospital-based cross-sectional study was conducted among 305 adult type 2 diabetic patients at public hospitals in Hadiya zone from March 1–30, 2019. The study participants were selected by systematic sampling technique. Data were collected using a pretested structured questionnaire and patient chart review; anthropometric and blood pressure measurements were taken. Multivariable logistic regression analysis was used to identify factors associated with poor glycemic control. Adjusted odds ratios (AOR) with respective 95% Confidence Interval (CI) and p < 0.05 were used to set statistically significant variables. Results Out of 305 diabetic patients, 222 (72.8%) were found to have poor glycemic control. Longer duration of diabetes (5–10 years) [AOR = 2.24, 95% CI: 1.17–4.27], lack of regular follow-up [AOR = 2.89, 95% CI: 1.08–7.71], low treatment adherence [AOR = 4.12, 95% CI: 1.20–8.70], use of other alternative treatments [AOR = 3.58, 95% CI: 1.24–10.36], unsatisfactory patient physician relationship [AOR = 2.27, 95% CI: 1.27–4.04], and insufficient physical activity [AOR = 4.14, 95% CI: 2.07–8.28] were found to be independent predictors of poor glycemic control. Diabetes Mellitus (DM) complications were slightly higher among participants with poor glycemic control (39.2%), duration of DM 10 and above years (41.9%), low medication adherence (48.5%), taking oral anti-diabetics (54.3%), and DM patients having unsatisfactory patient provider relationship (72.4%). Conclusion A significant proportion of diabetic patients had poor glycemic control and DM complications. Therefore, appropriate interventions are required to maintain optimal glycemic control and prevent the development of life-threatening complications among DM patients.