In cognitive radio networks (CRNs), improving system utility and ensuring system fairness are two important issues. In this paper, we propose a spectrum allocation model to construct CRNs based on graph coloring theory, which contains three classes of matrices: available matrix, utility matrix, and interference matrix. Based on the model, we formulate a system objective function by jointly considering two features: system utility and system fairness. Based on the proposed model and the objective problem, we develop an improved gravitational search algorithm (IGSA) from two aspects: first, we introduce the pattern search algorithm (PSA) to improve the global optimization ability of the original gravitational search algorithm (GSA); second, we design the Chebyshev chaotic sequences to enhance the convergence speed and precision of the algorithm. Simulation results demonstrate that the proposed algorithm achieves better performance than traditional methods in spectrum allocation.