Our previous results suggest that the Rosa rugosa Thunb. (family Rosaceae) alleviates endurance exercise-induced stress by decreasing oxidative stress levels. This study aimed to screen and identify the physiological antistress effects of an extract of R. rugosa (RO) on sleep deprivation-induced anxiety-like behavior and cognitive tests (in vivo) and tested for hippocampal CORT and monoamine levels (ex vivo), corticosterone (CORT)-induced injury, N-methyl-d-aspartate (NMDA) receptor, and serotonin 6 (5-hydroxytryptamine 6, 5-HT6) receptor activities (in vitro) in search of active principles and underlying mechanisms of action. We confirmed the antistress effects of RO in a sleep-deprived stress model in rat and explored the underlying mechanisms of its action. In conclusion, an R. rugosa extract showed efficacy and potential for use as an antistress therapy to treat sleep deprivation through its antagonism of the 5-HT6 receptor and resulting inhibition of cAMP activity.