The pathological features of Alzheimer’s Disease (AD) first appear in the medial temporal lobe and then in other brain structures with the development of the disease. In this work, we investigated the association between genetic loci and subcortical structure volumes of AD on 393 samples in the Alzheimer’s Disease Neuroimaging Initiative (ADNI) cohort. Brain subcortical structures were clustered into modules using Pearson’s correlation coefficient of volumes across all samples. Module volumes were used as quantitative traits to identify not only the main effect loci but also the interactive effect loci for each module. Thirty-five subcortical structures were clustered into five modules, each corresponding to a particular brain structure/area, including the limbic system (module I), the corpus callosum (module II), thalamus–cerebellum–brainstem–pallidum (module III), the basal ganglia neostriatum (module IV), and the ventricular system (module V). Gene Ontology (GO) and Kyoto Encyclopedia of Genes and Genomes (KEGG) enrichment results indicate that the gene annotations of the five modules were distinct, with few overlaps between different modules. We identified several main effect loci and interactive effect loci for each module. All these loci are related to the function of module structures and basic biological processes such as material transport and signal transduction.