How do verbal descriptions affect visual memory over the short-and long-term? Here we show for the first time that verbal labeling can boost visual memories, but the source of this benefit depends on whether representations are maintained over the short-term in visual working memory, or over the long-term in visual long-term memory. Across three experiments, we contrasted color memory of randomly colored objects when participants labeled (a) the color, (b) the object, or (c) the color-object binding, to memory under an articulatory suppression condition inhibiting labeling. Memory was tested at two time points: after three objects (visual working memory) and at the end of the experiment (visual long-term memory). In Experiment 1, color labeling improved, whereas object labeling impaired, visual working memory in comparison to suppression. Visual long-term memory remained unchanged across conditions. Experiment 2 tested whether this was due to poor overall long-term learning by repeating the colored objects over three successive working memory trials. This increased performance over the short and long-term; yet labeling did not change learning rate over repetitions or delayed memory performance, showing no long-term memory benefit. In Experiment 3, a labeling benefit was observed when the color-object binding was labeled both over the short-and long-term. Mixture modeling indicated that color-labeling benefits in visual working memory resulted from an increase of detailed visual memory, whereas long-term memory benefits accrued from categorical representations. Our findings point to dissociations on the role of language in visual working memory and visual long-term memory.