Smartwatches are the most popular wearable device and increasingly subject to empirical research. In recent years, the focus has shifted from revealing determinants of smartwatch adoption to understanding factors that cause long-term usage. Despite their importance for personal fitness, health monitoring, and for achieving health and fitness goals, extant research on the continuous use intention of smartwatches mostly disregards health and fitness factors. Grounding on self-determination theory, this study addresses this gap and investigates the impact of health and fitness as well as positive and negative emotional factors encouraging or impeding consumers to continuously use smartwatches. We build upon the expectation-confirmation model (ECM) and extend it with emotional (device annoyance and enjoyment) as well as health and fitness factors (goal pursuit motivation and self-quantification behavior). We use structural equation modeling to validate our model based on 335 responses from actual smartwatch users. Results prove the applicability of the ECM to the smartwatch context and highlight the importance of self-quantification as a focal construct for explaining goal pursuit motivation, perceived usefulness, confirmation and device annoyance. Further, we identify device annoyance as an important barrier to continuous smartwatch use. Based on our results, we finally derive implications for researchers and practitioners alike.