Visual working memory refers to the ability to temporarily hold information in mind in the service of behavior. Often, it is not sufficient to hold an abstract idea in mind to achieve our goals. Rather, we must maintain vivid sensory details. For example, when buying a spool of thread to repair a much-loved shirt, holding an abstract category in mind is not sufficient to buy the correct color (e.g. ‘blue’)—instead, you need a precise visual memory of the color (e.g., a particular gray-ish shade of blue). One proposal for how we maintain vivid, detailed information in mind is the sensory recruitment hypothesis. Sensory recruitment proposes that neural circuits already specialized for encoding sensory details during perception are likewise recruited to help maintain this information in working memory. In this review, we recount evidence that is consistent with a key role for early visual cortex in supporting visual working memory, we discuss key debates about the role of early sensory activity in supporting memory maintenance, and we outline a framework in which sensory codes are one part of a flexible, multi-level working memory representation.