Background Cancer can cause physical changes and affect satisfaction with a persons’ physical appearance, which in turn can affect overall quality of life. Previous studies have primarily focused on women with breast cancer and few is known about body image in patients with other cancers and especially men. The present study compares satisfaction with body image of patients with different types of cancer with the general population and across sexes and identifies risk factors for diminished body image. Additionally, patients that were diagnosed within the last year and those living with cancer for longer are compared. Methods In this cross-sectional study, N = 531 cancer patients answered the German Self-Image Scale to assess body image. One sample t-tests are utilized to compare the body image of cancer patients with the general population. Stepwise regression analyses were used to identify factors associated with body image and ANOVAs with posthoc tests as well as t-tests were used to examine group differences. Results Cancer patients showed diminished body image compared to the general population. For men, higher relationship satisfaction and lower cancer-specific distress were associated with more positive body self-acceptance (SA), whereas younger age, higher relationship satisfaction, and lower cancer-specific distress resulted in better perceived partner-acceptance of one’s body (PA). In women, higher education, lower anxiety and cancer-specific distress were associated with more positive SA. Female cancer patients with breast/gynecological cancer reported better SA than those with visceral cancers. Higher relationship satisfaction and lower cancer-specific distress were found to be associated with more satisfactory PA in females. Time since diagnosis did not affect body image in this study. Conclusions Results indicate that cancer patients regardless of sex tend to have decreased body image satisfaction. Future research directions include examination of additional entities of cancer, deeper research in men and the role of time since diagnosis.