Background: Changes in olfactory recognition memory may constitute sensory markers in depression. Significant differences may exist between unipolar and bipolar depression. Our study compares olfactory memory between control, unipolar, and bipolar patients in depressed and euthymic states in order to identify potential markers of depression. Methods: 176 participants were recruited in 5 groups: depressed bipolar (DB), euthymic bipolar (EB), depressed unipolar (DU), euthymic unipolar (EU), and controls (HC). The participants had a standardized clinical and olfactory assessment (olfactory memory, evaluation of pleasantness, intensity, familiarity, and emotional aspect of smells). Results: DU, DB, and EU patients had a deficit in olfactory memory compared to HC. DB patients had lower capacity to recognize new odors. DB and DU patients had more limited detection of unfamiliar odors than HC. DB patients rated odors as less pleasant compared to the other groups. All groups had lower hedonic ratings than HC. DB patients had lower emotional ratings than EU patients. Conclusions: Olfactory memory is impaired in depressive states, thus constituting a state marker of depression. Impairments in olfactory memory persist after remission of bipolar depression, thus constituting a possible trait marker of bipolarity. Hedonic rating differentiates unipolar from bipolar depression. This is the first study that identifies a sensory marker differentiating between unipolar and bipolar depression.