Depression-induced cognitive impairment has recently been given more attention in research. However, the relationship between depression and different types of memory is still not clear. Chronic unpredictable mild stress (CUMS) is a commonly used animal model of depression in which animals are exposed to chronic unpredictable environmental and psychological stressors, which mimics daily human life stressors. This study investigated the impact of different durations of CUMS on various types of memory (short- and long-term spatial memory and recognition memory) and investigated CUMS’ impact on the ultrastructural level by histological assessment of the hippocampus and prefrontal cortex. Twenty male C57BL/J6 mice (6 weeks old, 21.8 ± 2 g) were randomly divided into two groups (n = 10): control and CUMS (8 weeks). A series of behavioral tasks were conducted twice at weeks 5–6 (early CUMS) and weeks 7–8 (late CUMS). A tail-suspension test (TST), forced swimming test (FST), elevated zero maze (EZM), elevated plus maze (EPM), open field test (OFT), and sucrose-preference test (SPT) were used to assess anxiety and depressive symptoms. The cognitive function was assessed by the novel object recognition test (NORT; for recognition memory), Y-maze (for short-term spatial memory), and Morris water maze (MWM: for long-term spatial memory) with a probe test (for reference memory). Our data showed that 8 weeks of CUMS increased the anxiety level, reported by a significant increase in anxiety index in both EPM and EZM and a significant decrease in central preference in OFT, and depression was reported by a significant increase in immobility in the TST and FST and sucrose preference in the SPT. Investigating the impact of CUMS on various types of memory, we found that reference memory is the first memory to be affected in early CUMS. In late CUMS, all types of memory were impaired, and this was consistent with the abnormal histological features of the memory-related areas in the brain (hippocampus and prefrontal cortex).