Since the global onset of COVID-19 in early 2020, the disease has significantly impacted mental health. This impact is likely to be further exacerbated for groups who were already marginalized. This paper shares results from a broader study of men who have sex with men (MSM) and transgender people in Bali, Indonesia and includes a focus on psychological distress and happiness during the COVID-19 pandemic; applying sociodemographic and epidemiological characteristics as potential mediators. Psychological distress and the level of happiness were measured by The Kessler Psychological Distress (K10) and the Subjective Happiness Scale (SHS). A cross-sectional survey was conducted from July to September 2020. Of the 416 participants, complete data were available for 363 participants. The majority of participants were aged 26–40 years, currently single, were born outside Bali, were currently living in an urban area, and over one-third were living with HIV. While all were MSM, the majority identified as homosexual/tend to be homosexual (71.3%), however 54 (14.9%) identified themselves as heterosexual. The majority (251, 69.1%) reported moderate to very high psychological distress during the COVID-19 pandemic. The binary logistic regression analysis identified five factors to be significantly associated with higher psychological distress: being a student, reporting higher levels of stigma, had ever experienced discrimination, felt better prior to the COVID-19 pandemic, and less happy than the average person. When homosexual were compared with heterosexual participants, those who identified themselves as being homosexual reported significantly lower psychological distress compared to those identified themselves as heterosexual, which may be associated with these participants not disclosing their status as MSM and the stigma around MSM. Those who considered themselves to be less happy than the average person (316, 87.1%) were more likely to live with a partner and to report moderate to very high psychological distress. Based on the findings, interventions should focus on strategies to reduce stigma, provide non-discriminatory services, and improve access to essential health services.