The position and number of hair whorls have been associated with the behavior, temperament, and laterality of horses. The easy observation of whorls assists in the prediction of reactivity, and thus permits the development of better measures of handling, training, mounting, and riding horses. However, little is known about the genetics involved in the formation of hair whorls. Therefore, the aim of this study was to perform a genome-wide association analysis to identify chromosome regions and candidate genes associated with hair whorl traits. Data from 342 Quarter Horses genotyped for approximately 53,000 SNPs were used in an association study using a single-step procedure. The following traits were analyzed: vertical position of hair whorl on the head, number of whorls on the head, and number of whorls on the left and right sides of the neck. The traits had between one and three genomic windows associated. Each of them explained at least 4% of the additive variance. The windows accounted for 20–80% of additive variance for each trait analyzed. Many of the prospected genes are related to hair follicle growth. Some of these genes exert a pleiotropic effect on neurological and behavioral traits. This is the first indication of biological and physiological activity that might explain the association of hair whorls and temperament.