Site and climate conditions are the key determinants controlling dominant height growth and forest productivity, both independently and interactively. Secondary natural oak forests are a typical forest type in China, especially in Hunan Province, but little is known about the site index of this forest under the complex site and climate variables in the subtropics. Based on survey data of dominant trees and site variables from 101 plots in Hunan oak natural secondary forests and climate data obtained using spatial interpolation, we used the random forest method, correlation analysis, and the analysis of variance to determine the main site and climate factors affecting oak forest dominant height and proposed a modeling method of an oak natural secondary forest site index based on the random effect of site–climate interaction type. Of the site variables, elevation affected stand dominant height the most, followed by slope direction and position. Winter precipitation and summer mean maximum temperature had the greatest impact on stand dominant height. To develop the modeling method, we created 10 popular base models but found low performance (R2 ranged from 0.1731 to 0.2030). The optimal base model was Mitscherlich form M3 (R2 = 0.1940) based on parameter significance tests. Since site and climate factors affect the site index curve, the dominant site and climate factors were combined into site types and climate types, respectively, and a nonlinear mixed-effects approach was used to simulate different site types, climate types, site–climate interaction types, and their combinations as random effects. Site–climate interaction type as a random factor enhanced model (M3.4) performance and prediction accuracy (R2 from 0.1940 to 0.8220) compared to the optimum base model. After clustering the 62 site–climate interaction types into three, five, and eight groups using hierarchical clustering, a mixed-effects model with the random effects of eight groups improved model performance (R2 = 0.8265) and applicability. The modeling method developed in this study could be used to assess a regional secondary natural oak forest site index under complex site and climate variables to evaluate the forest productivity.