Background Personal logbooks are universally applied for monitoring and evaluation of surgical trainees; however, the quality and accuracy of such logbooks in low income countries (LICs) are poorly examined. Logbooks are kept by the individual trainee and detail every surgical procedure they perform and their role during the procedure. The aim of this study was to evaluate the quality of such a logbook system in Sierra Leone and to identify areas of improvement. Methods The last 100 logbook entries for students and graduates participating in a surgical task sharing training programme were compared with hospital records (HRs). The logbook entries were categorized as matching , close matching or over-reported . Moreover, HRs were checked for under-reported procedures. Semi-structured interviews were conducted with the study participants on logbook recording routines. The results were analysed using mixed effects logistic regression models. Results Three thousand one hundred sixty-nine database entries from 35 participants were analysed. Of that amount, 62.2% of the entries matched the HRs, 10.4% were close matches and 26.9% were over-reported. 20.7% of the investigated HRs were under-reported. Conclusions Information gathered from surgical logbook systems must be applied with care, and great efforts must be made to ensure that the logbook systems used provide reliable data. Based on analysis of the logbook data and interviews, focus areas are suggested to ensure reliable logbook data in LICs. Clear instructions and proper training should be provided when introducing the logbook system to the users. The importance of logging all procedures, including minor ones, should be emphasized. The logbook system should be user friendly and only as extensive as necessary. Lastly, keeping the logbooks exclusively digital is recommended, combined with sufficient IT equipment and training. Electronic supplementary material The online version of this article (10.1186/s12909-019-1647-2) contains supplementary material, which is available to authorized users.