Cytology is a simple and non-invasive screening method for oral cancer. However, this method is not yet routinely used by clinicians because of its high false negative rate (FNR) and due to lack of sufficient studies examining the factors for high FNRs. The present retrospective study aimed to compare the screening performance of conventional cytology (CC) and liquid-based cytology (LBC) through histological validation, and to elucidate factors inducing false negative screening in oral cytology. Cytological specimens with histological examination and intraoral digital images of the lesion were retrospectively collected between January 2017 and December 2018 for CC and between October 2019 and September 2021 for LBC. Oral cytological screening was conducted based on the oral Bethesda system for oral cytology. Clinical subtypes were re-evaluated using intraoral digital images. The screening accuracy of oral cytology was calculated considering the sensitivity, specificity, positive predictive value (PPV) and negative predictive value (NPV) for detecting the malignant transformation of oral lesions. No statistically significant difference was noted in the inadequate rate between CC and LBC groups. For CC and LBC, the sensitivities were 60.9 and 59.2%, the specificities were 87.3 and 79.1%, the PPVs were 85.8 and 76.2%, and the NPVs were 63.9 and 63.2%, respectively. Thus, the screening accuracy was similar between methodologies. Among the clinicopathological factors investigated, histological diagnosis and cellularity contributed to false negative results. Homogeneous findings of oral epithelial dysplasia and the superficial growth of carcinoma in situ/squamous cell carcinoma resulted in false negative findings for CC and LBC. Furthermore, LBC samples with a lower cell number (<2,000 squamous cells) exhibited statistically significantly increased FNRs. The present study found that the cytological methods did not affect the inadequate rate and screening accuracy, whereas clinical subtype and cellularity decreased screening accuracy. Therefore, cytological screening and subsequent follow-up should be performed while considering clinical findings and the cellularity of cytology smears.