A suitable self‐thinning model is fundamental to effective density control and management. Using data from 265 plot measurements in oak mixed forests in central China, we demonstrated how to estimate a suitable self‐thinning line for oak mixed forests from three aspects, i.e., self‐thinning models (Reineke's model and the variable density model), statistical methods (quantile regression and stochastic frontier analysis), and the variables affecting stands (topography and stand structure factors). The proposed variable density model, which is based on the quadratic mean diameter and dominant height, exhibited a better goodness of fit and biological relevance than Reineke's model for modeling the self‐thinning line for mixed oak forests. In addition, the normal‐truncated normal stochastic frontier model was superior to quantile regression for modeling the self‐thinning line. The altitude, Simpson index, and dominant height–diameter ratio (Hd$$ {H}_{\mathrm{d}} $$/D) also had significant effects on the density of mixed forests. Overall, a variable density self‐thinning model may be constructed using stochastic frontier analysis for oak mixed forests while considering the effects of site quality and stand structure on density. The findings may contribute to a more accurate density management map for mixed forests.