We developed a web-based tool to provide students with access to old exam problems and solutions. By controlling the order in which students saw the problems, as well as their access to solutions, we obtained data about student learning by studying old exam problems. Our data suggest that in general students learn from doing old exam problems, and that having access to the problem solutions increases their learning. However, the data also suggest the depth of learning may be relatively shallow. In addition, the data show that doing old exam problems provides important formative assessment about the student’s overall preparedness for the exam and their particular areas of strength and weakness