Goat milk yogurt infused with hemp extract (HE) is a novel dairy product; however, the unpleasant flavors from hemp terpenes and goat milk may impact its acceptance and popularity. This study aimed to investigate the effect of polymerized whey protein (PWP) on mitigating the hempy flavor of HE-infused goat milk yogurt and its impact on the physicochemical properties, microstructure, and probiotic survivability. Goat milk yogurt samples were infused with either nothing (plain flavor), HE, HE plus whey protein isolate, or HE plus PWP. Compared with plain goat milk yogurt, the addition of PWP in HE goat milk yogurt greatly improved the viscosity. The sensory evaluation results (N = 19) indicated that PWP significantly improved the consistency and decreased the hempy flavor of HE goat milk yogurt, although there was no difference in consumer acceptance. The microstructure analysis revealed that adding PWP formed a compact gel network compared to the irregular open protein matrixes in other groups. In conclusion, PWP not only improved the consistency of goat milk yogurt but was also useful in mitigating the hempy flavors of HE-infused goat milk.