The growing market of wearables is expanding into different areas of application such as devices designed to improve and monitor sport activities. This in turn is pushing research on low-cost, very low-power wearable systems with increased analysis capabilities. This paper proposes integrated energy-aware techniques and a convolutional neural network (CNN) for a cardiac arrhythmia detection system that can be worn during sport training sessions. The dynamic power management strategy (DPMS) is programmed into an ultra-low-power microcontroller, and in combination with a photovoltaic (PV) energy harvesting (EH) circuit, achieves a battery-life extension towards a self-powered operation. The CNN-based analysis filters, scales the image, and using a bicubic technique, interpolates the measurements to subsequently classify the electrocardiogram (ECG) signal into normal and abnormal patterns. Experimental results show that the EH-DPMS achieves an extension in the battery charge for a total of 14.34% more energy available, which represents 12 consecutive workouts of 45 min without the need to manually recharge it. Furthermore, an arrhythmia detection precision of 98.6% is achieved among the experimental sessions using 55,222 images for training the system with the MIT-BIH, QT, and long-term ST databases, and 1320 implemented on a wearable system. Therefore, the proposed wearable system can be used to monitor an athlete’s condition, reducing the risk of abnormal heart conditions during sports activities.