Introduction. Male infertility is known to contribute about half of all infertility cases. In Ghana, the prevalence of male infertility is higher (15.8%) than in females (11.8%). Sperm quality is associated with the likelihood of pregnancy and known to be the cause of male fertility problems 90% of the time. Exposure to certain environmental factors reduces semen quality in men. The study examined the effects of environmental and lifestyle factors on semen quality in Ghanaian men. Materials and Methods. This was a cross-sectional study involving 80 apparent healthy adult males in their reproductive age. Participants were males referred to the laboratory (Immunology Unit of the Korle-Bu Teaching Hospital) for semen analysis test and/or culture and sensitivity. Participants were made to fill out a questionnaire which entailed selected environmental factors (accidents or trauma, exposure to chemicals, radiation, and heat) and lifestyle habits (including alcohol consumption, smoking, and whether participants sat more or less than 4 hours per day). Semen samples were then collected by masturbation into sterile containers and analysed in accordance with WHO guidance for semen analysis within 60 minutes after ejaculation and collection. Results. About 69% of participants had semen pH within the normal range compared to 15% whose pH were lower than 7.2. There was a significantly high number of immotile sperm cells ( p value = 0.017) in participants who sat for more than 4 hours as compared to those that sat for less than 4 hours in a day. Active sperm motility and viability showed significant increase ( p value = 0.002 and 0.009, respectively) in participants who kept their cell phones in their side pockets. Smoking produced a twofold decrease in sperm count as smokers had a significantly lower sperm count ( 12.28 ± 10.95 × 10 6 /ml) compared to the smoke-free ( 23.85 ± 22.14 × 10 6 /ml). For exposure to STDs, no significant differences were recorded among study groups concerning semen quality. Conclusion. Sperm quality in Ghanaian men is associated with lifestyle habits. Smoking and sitting for long hours influenced sperm motility and count, respectively. Knowledge of the factors that influence sperm quality in this geographical region can contribute to informed decisions on effective management of infertility in Ghanaian men.