Milk production is influenced by many factors, including genetic and environmental factors and their interactions. Animal health, especially udder health, is usually evaluated by the number of somatic cells. The present study described the effect of polymorphisms in the ACACA, BTN1A1, LPL, and SCD genes on the daily milk yield, fat, and protein percentages and somatic cell count. In this study, 590 White Shorthaired (WSH) and Brown Shorthaired (BSH) goats were included. SNP genotyping was performed by PCR-RFLP and multiplex PCR followed by SNaPshot minisequencing analysis. The linear mixed model with repeated measurement was used to identify the genetic associations between the studied genes/SNPs and chosen traits. All selected genes were polymorphic in the tested goat populations and showed significant associations with milk traits. Only BTN1A1 (SNP g.599 A > G) showed a significant association with the somatic cell score. After Bonferroni correction, a significant effect of LPL g.300G > A on daily milk yield and fat percentage, LPL g.185G > T on protein percentage, and LPL G50C, SCD EX3_15G > A, and SCD EX3_68A > G on fat percentage was found. The importance of environmental factors, such as the herd-year effect, month of milking, and lactation order on all milk performance indicators was confirmed.