Maize plant height related traits including plant height, ear height, and internode number are tightly linked with biomass, planting density, and grain yield in the field. Previous studies have focused on understanding the genetic basis of plant architecture traits per se, but the genetic basis of combining ability remains poorly understood. In this study, 328 recombinant inbred lines were inter-group crossed with two testers to produce 656 hybrids using the North Carolina II mating design. Both of the parental lines and hybrids were evaluated in two summer maize-growing regions of China in 2015 and 2016. QTL mapping highlighted that 7 out of 16 QTL detected for RILs per se could be simultaneously detected for general combining ability (GCA) effects, suggesting that GCA effects and the traits were genetically controlled by different sets of loci. Among the 35 QTL identified for hybrid performance, 57.1% and 28.5% QTL overlapped with additive/GCA and non-additive/SCA effects, suggesting that the small percentage of hybrid variance due to SCA effects in our design. Two QTL hotspots, located on chromosomes 5 and 10 and including the qPH5-1 and qPH10 loci, were validated for plant height related traits by Ye478 derivatives. Notably, the qPH5-1 locus could simultaneously affect the RILs per se and GCA effects while the qPH10, a major QTL (PVE > 10%) with pleiotropic effects, only affected the GCA effects. These results provide evidence that more attention should be focused on loci that influence combining ability directly in maize hybrid breeding.