Shelf-life prediction for fruits based on the visual inspection and with RGB imaging through external features becomes more pervasive in agriculture and food business. In the proposed architecture, to enhance the accuracy with low computational costs we focus on two challenging tasks of shelf life (remaining useful life) prediction: 1) detecting the intrinsic features like internal defects, bruises, texture, and color of the fruits; and 2) classification of fruits according to their remaining useful life. To accomplish these tasks, we use the thermal imaging technique as a baseline which is used as non-destructive approach to find the intrinsic values of fruits in terms of temperature parameter. Further to improve the classification tasks, we combine it with a transfer learning approach to forecast the shelf life of fruits. For this study, we have chosen „Kesar? (Mangifera Indica Linn cv. Kesar) mangoes and for the purpose of classification, our designed dataset images are categorized into 19 classes viz. RUL-1 (Remaining Useful Life-1) to RUL-18 (Remaining Useful Life-18) and No-Life as after harvesting, the storage span of „Kesar? is near about 19 days. A comparative analysis using SqueezeNet, ShuffleNet, and MobileNetv2 (which are prominent CNN based lightweight models) has been performed in this study. The empirical results show a highest achievable accuracy of 98.15±0.44% with an almost a double speedup in training the entire process by using thermal images.