Tall fescue (Festuca arundinacea Schreb.) is one of the most important cool-season perennial obligatory outcrossing forage grasses in the United States. The production and persistence of tall fescue is significantly affected by drought in the south-central United States. Shoot-specific endophyte (Epichloë coenophiala)-infected tall fescue showed superior performance under both biotic and abiotic stress conditions. We performed a genome-wide association analysis using clonal pairs of novel endophyte AR584-positive (EP) and endophyte-free (EF) tall fescue populations consisting of 205 genotypes to identify marker-trait associations (MTAs) that contribute to drought tolerance. The experiment was performed through November 2014 to June 2018 in the field, and phenotypic data were taken on plant height, plant spread, plant vigor, and dry biomass weight under natural summer conditions of sporadic drought. Genotyping-by-sequencing of the population generated 3,597 high quality single nucleotide polymorphisms (SNPs) for further analysis. We identified 26 putative drought responsive MTAs (17 specific to EP, eight specific to EF, and one in both EP and EF populations) and nine of them (i.e., V.ep_10, S.ef_12, V.ep_27, HSV.ef_31, S.ep_30, SV.ef_32, V.ep_68, V.ef_56, and H.ef_57) were identified within 0.5 Mb region in the tall fescue genome (44.5–44.7, 75.3–75.8, 77.5–77.9 and 143.7–144.2 Mb). Using 26 MTAs, 11 tall fescue genotypes were selected for subsequent study to develop EP and EF drought tolerant tall fescue populations. Ten orthologous genes (six for EP and four for EF population) were identified in Brachypodium genome as potential candidates for drought tolerance in tall fescue, which were also earlier reported for their involvement in abiotic stress tolerance. The MTAs and candidate genes identified in this study will be useful for marker-assisted selection in improving drought tolerance of tall fescue as well opening avenue for further drought study in tall fescue.