Sperm analysis is an initial step in the examination conducted to identify infertility cases in humans. One aspect of sperm analysis involves observing the movement of spermatozoa and determining whether it is normal or abnormal. Normal spermatozoa movement is characterized by progressive motion at an average speed of 20 µm/second, while abnormal movement includes slow or non-motile spermatozoa. Traditional methods can be employed to assess the normality or abnormality of sperm movement, but they have drawbacks such as time-consuming procedures and diverse results depending on the expertise of the examiner. On the other hand, utilizing Computer-Assisted Sperm Analysis (CASA) equipment provides consistent results, albeit at a relatively high cost. Therefore, this research proposes an alternative method for determining sperm movement abnormalities using the Gaussian Mixture Model (GMM) for background subtraction and a matching-based algorithm to track and analyze the formed trajectories, distinguishing between normal and abnormal sperm movement. Human spermatozoa in real-time are used, and their movements are recorded in video format using a bright field microscope. The testing results for determining sperm movement abnormalities based on the GMM method and matching-based algorithm were successful, particularly in videos recorded at 50 fps recording speed, 20 minutes of liquefaction time, and 40x microscope lens magnification. This condition exhibited the highest average accuracy, with a tracking accuracy of 77.3% and an average accuracy for determining sperm motility abnormalities of 87.7%. Therefore, the combined tracking of sperm movement based on the GMM method and matching-based algorithm can be utilized to identify abnormalities in the movement of human spermatozoa.