Scientists imagine constantly. They do this when generating research problems, designing experiments, interpreting data, troubleshooting, drafting papers and presentations, and giving feedback. But when and how do scientists learn how to use imagination? Across six years of ethnographic research, it has been found that advanced career scientists feel comfortable using and discussing imagination, while graduate and undergraduate students of science often do not. In addition, members of marginalized and vulnerable groups tend to express negative views about the strength of their own imaginations, and the general usefulness of imagination in science. After introducing these findings and discussing the typical relationship between a scientist and their imagination across a career, we argue that reducing the number or power of active imaginations in science is epistemically counterproductive, and suggest a number of ways to bring imagination back into science in a more inclusive way, especially through courses on imagination for scientists, role models, and exemplar-based learning.