A noninvasive imaging technique providing information about testicular dysfunction in testes with varicocele would be useful. The aim is to measure the apparent diffusion coefficient (ADC) and fractional anisotropy (FA) in testes of infertile men with varicocele and to assess interobserver agreement. Sixteen infertile men with varicocele and 14 age‐matched controls underwent 1.5 T diffusion tensor imaging (DTI) MRI. Testicular ADC and FA were measured by two radiologists independently. Parametric and nonparametric statistical tests were applied to compare between the ADC and FA of testes with varicocele and normal testes. Interobserver agreement was evaluated. The interobserver variability for ADC (0.915) and FA (0.948) was very good. No differences in ADC (p = 0.294) were found between the two groups. FA was significantly lower in testes with varicocele compared to age‐matched controls (p < 0.001). An optimal cut‐off of FA 0.08 was found for the diagnosis of varicocele (sensitivity = 88%, specificity = 93.5%, positive predictive value = 91.6% and negative predictive value = 90.6%). Based on our results, FA is useful for the diagnosis of testes in infertile men with varicocele, with very good interobserver agreement. Therefore, DTI may be used as a noninvasive imaging tool in the work‐up of varicocele.