Objectives: This study aimed to determine postpartum modern contraceptive use among first-time young mothers attending child welfare clinics in the Eastern Region of Ghana and explore factors that influence family planning uptake after the first delivery, including fear of infertility. Methods: This facility-based, cross-sectional study used interviewer-administered structured questionnaires. The study recruited 422 first-time young mothers aged 15–24 years, with 6- to 18-month-old babies attending child welfare clinics. Results: Overall, less than half (44%) of first-time mothers used modern contraceptives within 18 months after delivery. Fear of infertility after contraceptive use (56%) is the main barrier reported as the reason for women’s non-use of modern contraceptives. Mothers with tertiary education have higher odds of using postpartum contraceptives (adjusted odds ratio =1.6, 95% confidence interval: 0.4–2.0). Compared to mothers with younger children, those with children older than 6 months have higher odds of postpartum contraceptive use (adjusted odds ratio = 1.3, 95% confidence interval: 0.3–1.7). Nonspousal communication (adjusted odds ratio = 0.1, 95% confidence interval: 0.1–0.3) as compared to communication among partners about contraception and those in formal employment (adjusted odds ratio = 0.3, 95% confidence interval: 0.1–0.7), were less likely to use postpartum contraceptives. Conclusion: Considering that there is low postpartum contraceptive utilization, mostly due to concerns about fear of infertility after use, it is paramount to intensify education on actual side effects and reformulate policies that address specific concerns of infertility among mothers and contraceptive use.