Relatively high-latitude coral reefs could be potential “refuges” for corals under climate change. One of the most important aspects limiting their availability as refuges is low-temperature stress. However, the mechanisms underlying the response of coral holobionts to low-temperature stress is unclear. In this study, we aimed to explore the underlying mechanisms by recording the maximum quantum yields of photosystem II (Fv/Fm) and transcriptome responses of Porites lutea under acute (1–2 weeks) and chronic (6–12 weeks) low-temperature stress at 20°C and 14°C. The P. lutea samples were collected from a relatively high-latitude coral reef in the South China Sea (109°00′–109°15′E and 21°00′–21°10′ N). The study suggested that: (1) Under acute low-temperature stress, the Fv/Fm of Symbiodiniaceae dropped by 64%, which was significantly higher than the 49% observed under chronic stress. Low-temperature stress inhibited photosystem II(PSII) functioning, with greater inhibition under acute stress. (2) Downregulation of sugar metabolism-related genes under low-temperature stress implied that the decrease in energy was due to obstruction of PSII. (3) Under low-temperature stress, calcification-related genes were downregulated in coral hosts, possibly because of energy deprivation caused by inhibited photosynthesis, Symbiodiniaceae expulsion, and oxidative phosphorylation uncoupling in mitochondria. (4) Acute low-temperature stress induced the upregulation of genes related to the TNF signaling pathway and endoplasmic reticulum stress, promoting apoptosis and coral bleaching. However, these phenomena were not observed during chronic stress, suggesting acclimation to chronic low-temperature stress and a greater survival pressure of acute low-temperature stress on coral holobionts. In conclusion, low-temperature stress inhibits Symbiodiniaceae PSII functioning, reducing energy production and affecting calcification in coral holobionts. Acute low-temperature stress is more threatening to coral holobionts than chronic stress.