Public speaking occupies a central position in secondary education, and often teachers introduce improvisation training to improve this competence, but it is yet unclear if this teaching approach has a beneficial impact. In this respect, this paper addresses the following research question: Does an improvisation training that focuses on eye-contact, voice, and body language have an impact on the public speaking competence of students in secondary education? In this study with a quasi-experimental design, we assessed the impact of the intervention with a control group with a pre-and posttest of a speaker's public speaking competence scored via compared judgement of videos of speaking exercises. The intervention had a significant but modest effect, but a number of important caveats need to be formulated. Age of participants, group size, and professionalism of the instructor might all partly determine the effect of such an improvisation training, but these variables were not integrated into the research design of this study.