[This paper is part of the Focused Collection on Astronomy Education Research.] Students taking a second astronomy course for nonscientists were asked to reflect on the nature of scientific inquiry three times during the first half of the semester. First, they were assigned a short paper in which they were asked to argue for or against the thesis that observers of the night sky in ancient civilizations were scientists, and nearly all of the students argued that yes, they were. Second, they were asked to write a second paper in which they argued the opposite of their first thesis. Lastly, they were asked, on a midterm exam, to write an essay about whether the Ancient Greek astronomers were good scientists. In writing these papers and essays, many students included definitions of science or scientists, and these definitions progressed from paper to paper. In addition, the evidence students used to argue that ancient observers were scientists changed, with students becoming more expertlike.