Cervical cancer is a severe and pervasive disease that poses a significant health threat to women globally. The Pap smear test is an efficient and effective method for detecting cervical cancer in its early stages. However, manual screening is labor-intensive and requires expert cytologists, leading to potential delays and inconsistencies in diagnosis. Deep Learning-based Computer-Aided Diagnosis (CAD) has shown significant results and can ease the problem of manual screening. However, one single model is sometimes insufficient to capture the complex data pattern for accurate disease prediction. In this work, we develop an end-to-end architecture utilizing three pre-trained models for the initial cervical cancer prediction. To aggregate the outcomes of these models, we propose a novel fuzzy rank-based ensemble considering two non-linear functions for the final level prediction. Unlike simple fusion techniques, the proposed architecture provides the final predictions on the test samples by considering the base classifier's confidence in the predictions. To further enhance the classification capabilities of these models, we integrate advanced augmentation techniques such as CutOut, MixUp, and CutMix. The proposed model is evaluated on two benchmark datasets, SIPaKMeD, and Mendeley LBC, using a 5-fold cross-validation approach. On the SIPaKMeD dataset, the proposed ensemble architecture achieves a classification accuracy of 97.18% and an F1-score of 97.16%. On the Mendeley LBC dataset, the accuracy reaches 99.22% with an F1-score of 99.19%. Experimental results demonstrate the proposed architecture's effectiveness and potential in cervical Pap smear image classification. This could aid medical professionals in making more informed treatment decisions, improving overall effectiveness in the testing process.