Flowering and grain-filling stages are highly sensitive to heat and drought stress exposure, leading to significant loss in crop yields. Therefore, phenotyping to enhance resilience to these abiotic stresses is critical for sustaining genetic gains in crop improvement programs. However, traditional methods for screening traits related to these stresses are slow, laborious, and often expensive. Remote sensing provides opportunities to introduce low-cost, less-biased, high-throughput phenotyping methods to capture large genetic diversity to facilitate enhancement of stress resilience in crops. This review focuses on four key physiological traits or processes that are critical in understanding crop responses to drought and heat stress during reproductive and grain-filling periods. Specifically, these traits include: i) time-of-day of flowering, to escape these stresses during flowering, ii) optimizing photosynthetic efficiency, iii) storage and translocation of water-soluble carbohydrates, and iv) yield and yield components to provide in-season yield estimates. An overview of current advances in remote sensing in capturing these traits, limitations with existing technology and future direction of research to develop high-throughput phenotyping approaches for these traits are discussed in this review. In the future, phenotyping these complex traits will require sensor advancement, high-quality imagery combined with machine learning methods, and efforts in transdisciplinary science to foster integration across disciplines.