There is an urgent need for generalized training in cardiopulmonary resuscitation (CPR) techniques, starting with secondary education. Validated instruments for assessing the efficacy of such interventions are not yet available. This study aimed to validate an evaluation questionnaire of a CPR training program for high school students, to analyze the levels of readability, difficulty, reliability, and content validity, as well as the fit the purpose for which they were designed, the trait they are intended to measure. An instrumental study was conducted in 2 phases. In the first phase, an inter-judge validation was carried out with 11 experts in CPR accredited instructors of basic and advanced CPR by the American Heart Association. In the second phase, the psychometric properties were evaluated from the perspective of Item Response Theory. During May of the 2017/18 and 2018/19 academic years, 259 4th-year secondary school students from a high school in the southeastern area of the island of Gran Canaria (mean age: 15.78 years; 50.60% male) were surveyed anonymously using the questionnaire to be validated. The questionnaire was easily readable (74.12 Flesch-Szigriszt Index); the difficulty level (Easy/Very Easy) in the context of this secondary school level of education and the ability level of the respondents overlapped sufficiently and there was no gender bias. The questionnaire was able to discriminate between respondents of slightly more than 7 levels of expertise, from low knowledge of CPR to high knowledge of CPR (Separation Index 7.53). The model fit was excellent (infit = 1/outfit = 1.01). The content validity index was adequate. The separation index and reliability exceeded what was considered adequate for guaranteed use. The level of difficulty of the items and the level of ability of the respondents is in line with the educational level of the students. The questionnaire did not produce a gender bias in response probability. The questionnaire is easily understandable and can discriminate between different levels of ability without differential gender bias, and its reliability is outstanding, as it exceeds the minimum criteria.