Although mastectomy could lead to a decrease in sexual performance among patients, only a handful of studies focused on the psychological and sexual behavioral aspects after the surgery. Research on post-mastectomy sexuality has focused mainly on female subjects but barely on lesbian, gay, bisexual, transgender, queer (LGBTQ), and male patients. This narrative review aimed to explore the importance of sexuality after mastectomy from a LGBTQ perspective. Each sexual minority group has been addressed individually. In general, sexual and gender minority breast cancer (BC) patients undergoing bilateral mastectomy expect a complex treatment plan in terms of physical and emotional outcomes. Bilateral mastectomy or top surgery for masculinization reasons was reported to be the most popular procedure among transmen, which resulted in a significant improvement in the quality of life. Heterosexual and lesbian female patients are willing to undergo mastectomy after repeated lumpectomies or to avoid radiation, despite potential post-operative somatic and quality-of-life complications. Transwomen would seek gender-affirming surgery to improve physical satisfaction and psychological well-being. There is not enough evidence for non-oncological reasons and consequences of mastectomy in gay men and cisgender heterosexual men. Establishing the awareness of the sexuality impact of mastectomy will allow the implementation of tailored perioperative psychological care.