Humans have been exploring colors since ancient times, but relatively complete color systems appeared one after another in the twentieth century. Even without language and other information exchanges, colors can still convey information and stimulate emotions. Therefore, color can have both physical and psychological effects on people. In this context, this paper studies the visual representation of painting colors based on psychological factors. The article studies the theory of personality traits and introduces the related content of visual psychology. To explore the relationship between each variable and color psychology and the visual representation of painting colors, a binary logistic regression analysis is performed. The colors in the post-impressionist paintings of Van Gogh and Gauguin is contrasted, and experiments on psychological factors and color research is conducted. The factors that affect the color tone of the picture and the influence of psychological factors on the judgment of color brightness are investigated. Finally, the correlation analysis of personality trait dimension and irrational behavior is carried out. The experimental results of the article show that after the analysis of variance, the significance levels of regression model 1 and model 2 both reach 0.000, and the adjusted R squares are 0.319 and 0.356, respectively. In this study, regression model 2 was selected as the final model. According to Model 2, the standardized regression coefficients of agreeableness and neuroticism are 0.438 and -0.251, respectively, and the significance of the regression coefficients are 0.000 and 0.021, respectively. The research on the visual performance of painting colors based on psychological factors has been well completed.