Background: People who identify as sexual and gender minorities (SGM) experienced disproportionate economic and mental health issues related to COVID-19 when compared to the general population. The purpose of this study was to better understand how COVID-19 has impacted the SGM community and ways to address vaccine hesitancy. Methods: Three focus groups were conducted with 21 members of the SGM community between 5 November and 10 December 2020. A thematic analysis using the reflexive approach was applied to the transcripts of the focus groups. Results: Four themes emerged: (1) Impact of COVID-19 on the Community, (2) Perceptions of Contact Tracing and Testing, (3) Perceptions of a Potential COVID-19 Vaccine, and (4) Decreasing Vaccine Hesitancy. The most relevant subthemes were that social isolation led to anxiety, stress, and fear in the SGM community during COVID-19; resilience and adaptation were positive outcomes of the pandemic; histories of medical racism contributed to hesitancy to get tested; and specific messaging from trusted messengers may be needed to encourage SGM communities to get vaccinated. These findings support other COVID-19 research on the SGM community during the start of the pandemic. Conclusions: This study provides insight into the impact of the early stages of COVID-19 on the SGM community, highlighting the unique hurdles faced by SGM individuals with regard to contact tracing and vaccine hesitancy.