Image contrast enhancement uses the object intensity transformation function to maximize the amount of information to enhance an image. In this paper, the image enhancement problem is regarded as an optimization problem, and the particle swarm algorithm is used to obtain the optimal solution. First, an improved particle swarm optimization algorithm is proposed. In this algorithm, individual optimization, local optimization, and global optimization are used to adjust the particle’s flight direction. In local optimization, the topology is used to induce comparison and communication between particles. The sparse penalty term in speed update formula is added to adjust the sparsity of the algorithm and the size of the solution space. Second, the three channels of the color images R, G, and B are represented by a quaternion matrix, and an improved particle swarm algorithm is used to optimize the transformation parameters. Finally, contrast and brightness elements are added to the fitness function. The fitness function is used to guide the particle swarm optimization algorithm to optimize the parameters in the transformation function. This paper verifies via two experiments. First, improved particle swarm algorithm is simulated and tested. By comparing the average values of the four algorithms under the three types of 6 test functions, the average value is increased by at least 15 times in the single-peak 2 test functions: in the multi-peak and multi-peak fixed-dimension 4 test functions, this paper can always search for the global optimal solution, and the average value is either the same or at least 1.3 times higher. Second, the proposed algorithm is compared with other evolutionary algorithms to optimize contrast enhancement, select images in two different data sets, and calculate various evaluation indicators of different algorithms under different images. The optimal value is the algorithm in this paper, and the performance indicators are at least a 5% increase and a minimum 15% increase in algorithm running time. Final results show that the effects the proposed algorithm have obvious advantages in both subjective and qualitative aspects.