Brain asymmetry, or the structural and functional specialization of each brain hemisphere, has fascinated neuroscientists for over a century. Even so, genetic and environmental factors that influence brain asymmetry are largely unknown. Diffusion tensor imaging (DTI) now allows asymmetry to be studied at a microscopic scale by examining differences in fiber characteristics across hemispheres rather than differences in structure shapes and volumes. Here we analyzed 4 Tesla DTI scans from 374 healthy adults, including 60 monozygotic twin pairs, 45 same-sex dizygotic pairs, and 164 mixed-sex DZ twins and their siblings; mean age: 24.4 years +/− 1.9SD). All DTI scans were nonlinearly aligned to a geometrically-symmetric, population-based image template. We computed voxel-wise maps of significant asymmetries (left/right differences) for common diffusion measures that reflect fiber integrity (fractional and geodesic anisotropy; FA, GA and mean diffusivity, MD). In quantitative genetic models computed from all same-sex twin pairs (N=210 subjects), genetic factors accounted for 33% of the variance in asymmetry for the inferior fronto-occipital fasciculus, 37% for the anterior thalamic radiation, and 20% for the forceps major and uncinate fasciculus (all L>R). Shared environmental factors accounted for around 15% of the variance in asymmetry for the cortico-spinal tract (R>L) and about 10% for the forceps minor (L>R). Sex differences in asymmetry (men > women) were significant, and were greatest in regions with prominent FA asymmetries. These maps identify heritable DTI-derived features, and may empower genome-wide searches for genetic polymorphisms that influence brain asymmetry.