Early detection of sub-clinical mastitis via thermogram analysis by using a part of the milking machine without disturbing the dairy animal can be a milestone in precision farming. In the present study, thermal images of the short milking tube of milking machine representing four udder quarters independently attached to a milking animal, along with pre- and post-milking udder and teat thermograms were taken using a hand-held digital infrared thermal camera (DarviDTL007) during morning milking of lactating Murrah buffaloes (n = 132) to assess the sub-clinical mastitis status. California mastitis test (CMT) and somatic cell count (SCC) of milk samples were carried out to screen the udder quarters as healthy, sub-clinical (SCM), and clinical mastitis (CM). The thermograms revealed a significant increase (p < 0.05) of 2.19 and 3.72ºC in the mean values of short-milking-tube (SMT) surface temperature among SCM and CM quarters compared to healthy quarters, respectively. The mean values of udder skin surface temperature (USST) for pre-milking, milking, and post-milking of SCM and CM compared to healthy quarters showed significant increase (p < 0.05) of 2.17, 1.96 and 1.61ºC and 3.11, 2.88 and 2.73ºC, respectively. Similarly, the mean values of teat skin surface temperature (TSST) for pre-milking and post-milking of SCM and CM compared to healthy quarters showed an increase of 2.12 and 1.66ºC; and 3.07 and 2.45ºC, respectively. Also, CMT and SCC results showed strong positive correlation (r = 0.68–0.91, p < 0.01) with all the thermographic parameters. Thus, thermograms of SMT alone can be used as an efficient detection tool in assessing SCM among Murrah buffaloes.