This study aimed to identify the genes related to the body size of pigs by conducting genome-wide selection analysis (GWSA). We performed a GWSA scan on 50 pigs belonging to four small-bodied pig populations (Diannan small-eared pig, Bama Xiang pig, Wuzhishan pig, and Jeju black pig from South Korea) and 124 large-bodied pigs. We used the genetic parameters of the pairwise fixation index (FST) and π ratio (case/control) to screen candidate genome regions and genes related to body size. The results revealed 47,339,509 high-quality SNPs obtained from 174 individuals, while 280 interacting candidate regions were obtained from the top 1% signal windows of both parameters, along with 187 genes (e.g., ADCK4, AMDHD2, ASPN, ASS1, and ATP6V0C). The results of the candidate gene (CG) annotation showed that a series of CGs (e.g., MSTN, LTBP4, PDPK1, PKMYT1, ASS1, and STAT6) was enriched into the gene ontology terms. Moreover, molecular pathways, such as the PI3K-Akt, HIF-1, and AMPK signaling pathways, were verified to be related to body development. Overall, we identified a series of key genes that may be closely related to the body size of pigs, further elucidating the heredity basis of body shape determination in pigs and providing a theoretical reference for molecular breeding.