Immune checkpoint inhibitors, including those targeting the PD-1/PD-L1 and CTLA-4 pathways, are revolutionizing cancer therapeutics. Both activity and toxicities largely stem from unleashing tumor- or host-specific cytotoxic T cells. Many patients seen in routine clinical practice did not qualify for, or were seriously underrepresented in immune checkpoint inhibitor clinical trials. Thus, a major gap in knowledge regarding the safety and efficacy of these agents persists in many populations, even following regulatory approval. To address this challenge, we aggregated and synthesized the available pre-clinical and clinical data surrounding immune checkpoint inhibitor therapy in challenging clinical populations to assist treatment decision making for both academic and community oncologists. Specifically, we focus on the safety and activity of immune checkpoint inhibitors in patients with autoimmune disorders, post-organ transplant, chronic viral infections, ongoing immunosuppressant use, organ dysfunction, pregnancy, brain metastases, extremes of age, and impaired functional status.