In this paper, a new frequency domain behavior model for radio frequency (RF) power transistors based on a hyper‐parameter optimized Gaussian process regression (GPR) method is presented. The chaos game optimization (CGO) algorithm is used to optimize GPR hyperparameters, resulting in the CGO‐GPR model. The basic theory as well as the details of the modeling process are presented. Validation of the model is conducted using a 10‐watt GaN power transistor. Compared to the standard GPR model, the proposed model achieved a significant improvement. Furthermore, the comparison with the particle swarm optimization (PSO) based GPR model (PSO‐GPR) showed that the proposed model allows achieving superior performance, thereby confirming the effectiveness of the developed modeling technique.