Modern artificial intelligence (AI) tools built on high‐dimensional patient data are reshaping oncology care, helping to improve goal‐concordant care, decrease cancer mortality rates, and increase workflow efficiency and scope of care. However, data‐related concerns and human biases that seep into algorithms during development and post‐deployment phases affect performance in real‐world settings, limiting the utility and safety of AI technology in oncology clinics. To this end, the authors review the current potential and limitations of predictive AI for cancer diagnosis and prognostication as well as of generative AI, specifically modern chatbots, which interfaces with patients and clinicians. They conclude the review with a discussion on ongoing challenges and regulatory opportunities in the field.