Background and Objectives: Pain affects psychological stress and general health in the working population. However, the factors affecting psychological job stress related to chronic pain are unclear. This study aimed to clarify the structural differences among factors affecting psychological job stress in workers with chronic pain and those without pain. Materials and Methods: A stepwise multiple regression analysis revealed the differences in structure between the psychological stress of workers with chronic pain and those with no pain. Psychological job stress by the Brief Job Stress Questionnaire was used as the dependent variable, with psychological state (depression and anxiety), specifically that characteristic of chronic pain (pain catastrophizing); information on the nature of the pain (intensity and duration); and number of years of service as independent variables. Selected independent variables were evaluated for collinearity. Results: In the model with psychological stress as a dependent variable (chronic pain: r2 = 0.57, F = 41.7, p < 0.0001; no-pain: r2 = 0.63, F = 26.3, p < 0.0001), the difference between the experiences of workers with chronic pain and those with no pain was that chronic pain was associated with depression (Beta = 0.43, p < 0.0001) and no pain with anxiety (Beta = 0.34, p < 0.0001). In the model with chronic pain-related depression as a dependent variable (r2 = 0.62, F = 41.7, p < 0.0001), job-life satisfaction (Beta = −0.18, p = 0.0017) and magnification (a dimension of pain catastrophizing; Beta = 0.16, p < 0.0001) were significant. Conclusions: The results of this study suggest that the psychological characteristics of chronic pain, such as depression and magnification, should be considered when evaluating and intervening in the job stress of workers with chronic pain.