Background: Due to demographic changes, the need to prolong working life has become increasingly salient. Paradoxically, stereotyping and discrimination against workers based on their age can limit possibilities for a long working life. A large body of research has investigated attitudes towards older workers; however, less is known about differences across occupational groups as well gender differences. Aim: To compare perceptions of the managers’ perceptions of older workers between employees from mainly seated work and mainly physical work as well as men and women. Method: Data from 11,444 workers aged 50+ from the baseline questionnaire survey in the SeniorWorkingLife study were analyzed. Results: Across all groups, a larger proportion of the respondents indicated that their managers had more positive than negative perceptions of older workers. Respondents from the International Standard Classification of Occupations (ISCO) group 5–9 (mainly physical work) and women were less likely to point at both positive and negative perceptions than the ISCO group 14 (mainly seated work) and men, respectively. Conclusions: The results indicate that there are differences between those with mainly physical and mainly seated work as well as gender differences. More research is warranted to explore the nature of these differences and, in particular, attitudes towards older female and male workers, respectively.