In this study, we found that both heat and cold stresses significantly affected the survival and reproduction of both sexes in Spodoptera frugiperda adults, with larvae showing relatively higher extreme temperature tolerance. Further transcriptomic analysis in adults found remarkable differences and similarities between sexes in terms of temperature stress responses. Metabolism-related processes were suppressed in heat stressed females, which did not occur to the same extend in males. Moreover, both heat and cold stress reduced immune activities in both sexes. Heat stress induced the upregulation of many heat shock proteins in both sexes, whereas the response to cold stress was insignificant. More cold tolerance-related genes, such as cuticle proteins, UDP-glucuronosyltransferase, and facilitated trehalose transporter Tret1, were found upregulated in males, whereas most of these genes were downregulated in females. Moreover, a large number of fatty acid-related genes, such as fatty acid synthases and desaturases, were differentially expressed under heat and cold stresses in both sexes. Heat stress in females induced the upregulation of a large number of zinc finger proteins and reproduction-related genes; whereas cold stress induced downregulation in genes linked to reproduction. In addition, TRPA1-like encoding genes (which have functions involved in detecting temperature changes) and sex peptide receptor-like genes were found to be differentially expressed in stressed moths. These results indicate sex-specific heat and cold stress responses and adaptive mechanisms and suggest sex-specific trade-offs between stress-resistant progresses and fundamental metabolic processes as well as between survival and reproduction.