In the context of feature-based image registration, the crucial task of outlier removal plays a pivotal role in achieving precise registration accuracy. This research introduces an innovative binary classifier founded on an adaptive approach for effectively identifying and eliminating outliers. The methodology begins with the utilization of the scale invariant feature transform (SIFT) to extract features from two images, initially matched using the Euclidian distance metrics. Subsequently, a classification procedure is executed to segregate the feature points into two categories: genuine matches (inliers) and spurious matches (outliers), which is accomplished through the brute-force matcher (BFM) technique. To enhance this process further, a novel classifier rooted in the random forest algorithm is introduced. This classifier is trained and tested using a comprehensive dataset curated for this study. The newly proposed classifier plays a pivotal role in attenuating the influence of outliers, ultimately leading to refined image registration process characterized by enhanced accuracy. The effectiveness of this outlier removal approach is assessed through a meticulous analysis of positional and classification accuracy. Additionally, we offer comparative insights by evaluating the performance of selected algorithm on our dataset.