A large body of evidence shows that sugars function both as nutrients and signals to regulate fruit and seed set under normal and stress conditions including heat and drought. Inadequate sucrose import to, and its degradation within, reproductive organs cause fruit and seed abortion under heat and drought. As nutrients, sucrose-derived hexoses provide carbon skeletons and energy for growth and development of fruits and seeds. Sugar metabolism can also alleviate the impact of stress on fruit and seed through facilitating biosynthesis of heat shock proteins (Hsps) and non-enzymic antioxidants (e.g., glutathione, ascorbic acid), which collectively maintain the integrity of membranes and prevent programmed cell death (PCD) through protecting proteins and scavenging reactive oxygen species (ROS). In parallel, sugars (sucrose, glucose, and fructose), also exert signaling roles through cross-talk with hormone and ROS signaling pathways and by mediating cell division and PCD. At the same time, emerging data indicate that sugar-derived signaling systems, including trehalose-6 phosphate (T6P), sucrose non-fermenting related kinase-1 (SnRK), and the target of rapamycin (TOR) kinase complex also play important roles in regulating plant development through modulating nutrient and energy signaling and metabolic processes, especially under abiotic stresses where sugar availability is low. This review aims to evaluate recent progress of research on abiotic stress responses of reproductive organs focusing on roles of sugar metabolism and signaling and addressing the possible biochemical and molecular mechanism by which sugars regulate fruit and seed set under heat and drought.