As obesity is a serious problem in the human population, overloading of the horse’s thoracolumbar region often affects sport and school horses. The advances in using infrared thermography (IRT) to assess the horse’s back overload will shortly integrate the IRT-based rider-horse fit into everyday equine practice. This study aimed to evaluate the applicability of entropy measures to select the most informative measures and color components, and the accuracy of rider:horse bodyweight ratio detection. Twelve horses were ridden by each of the six riders assigned to the light, moderate, and heavy groups. Thermal images were taken pre- and post-exercise. For each thermal image, two-dimensional sample (SampEn), fuzzy (FuzzEn), permutation (PermEn), dispersion (DispEn), and distribution (DistEn) entropies were measured in the withers and the thoracic spine areas. Among 40 returned measures, 30 entropy measures were exercise-dependent, whereas 8 entropy measures were bodyweight ratio-dependent. Moreover, three entropy measures demonstrated similarities to entropy-related gray level co-occurrence matrix (GLCM) texture features, confirming the higher irregularity and complexity of thermal image texture when horses worked under heavy riders. An application of DispEn to red color components enables identification of the light and heavy rider groups with higher accuracy than the previously used entropy-related GLCM texture features.