Nonsomatic factors play a significant role in erectile dysfunction. We assessed the impact on men suffering from male factor infertility by employing the validated Self‐Esteem and Relationship questionnaire (SEAR) and compared the scores with controls. Men with primary infertility were asked to fill SEAR questionnaire. Normal fertile men visiting clinic for unrelated problem or normal male accompanying infertile patients constitute the control group. Transformed score of each domain and total score of SEAR questionnaire and means were compared. Data were analysed by R version 5.2. Univariate and multivariate analyses were done to determine factors predicting self‐esteem and total relationship score. There were 45 men each in the study and control groups. The mean transformed self‐esteem score and total score of infertile men were significantly lower as compared to controls (74.44 versus 95.83) and (73.54 versus 95.86) p‐value of <.0001. Longer duration of infertility, advanced age, diabetes and higher education were factors significantly lowering the scores on univariate analysis, and on multivariate analysis, diabetes was the only significant predictor of total SEAR score in infertile men. Infertility leads to lower self‐esteem, sexual performance and confidence among infertile men as compared to controls. The relationship score worsens with increasing duration of infertility.