Machine learning based fiber optic sensing technology is expected to achieve integrated low-cost demodulation solutions in future miniaturized human temperature sensing. In this work, we proposed a machine learning of speckle images assisted wearable temperature skin with implanted fiber optic sensor. The balloon type sensor is sandwiched between two flexible Polydimethylsiloxane (PDMS) films and nested on a hard UV polymer cap. The volume of UV polymer cap will change with temperature, thereby driving changes in the radius of balloon shaped optical fiber. This leads to changes in the speckle pattern generated at the end of the optical fiber, which is then demodulated through machine learning. The experiment shows the speckle variation of the sensor every 0.1°C in the temperature range of 36.4°C to 37.4°C. The "SpeckleNet" regression model based on Visual Geometry Group-16 (VGG-16) is proposed, which adopts fewer convolutional and max pooling kernels, and uses fewer fully-connected layers to reduce computational complexity. The prediction accuracy of the model can reach 99.88%. The research content of this article has good application prospects in the field of human wearable temperature sensors.