Introduction There is no consensus on the best methodology to apply evidence-based practice principles to develop a systematic approach to improve critical appraisal or research design evaluation skills in advanced education journal clubs. Methods We implemented a tool-based approach for our pediatric dentistry residents’ journal club centered on the use of a study quality assessment tool, the Timmer scale. The tool consisted of 19 standard questions that evaluated the research methodology, data collection, statistical analysis, and reporting of the findings of each article. Learners first underwent a 4-hour training session on study quality assessment. They were then assigned to read articles from monthly issues of core journals and appraised the quality of each article using the Timmer scale and submitted their scores in advance of the group session. Then, during a 1-hour journal club, the group came to a consensus on the Timmer scale score, and group and individual scores were compared to the course director's scores as prompts for feedback and further discussion. Results Over 3 years, 24 pediatric dentistry residents participated in the course. A noticeable improvement in the pediatric dentistry residents’ performance was noticed, with the discrepancy between their scores and group scores improving over time. Discussion Using a quality assessment tool in journal clubs appeared to improve the residents’ ability to critically assess articles in a systematic way. Additionally, the tool was useful for assessing residents’ performance over time.