(1) Background: Many studies have revealed a relationship between serum 25-hydroxy vitamin D and physical activity. This study aimed to investigate the relationship between self-reported sitting time and serum 25-hydroxy vitamin D levels in middle-aged and elderly adults in Taiwan. (2) Methods: A total of 396 people were enrolled in our study during a community health examination in Taiwan in 2019. We grouped participants from low to high according to their tertile of serum 25-hydroxy vitamin D levels, using the following categories: deficiency, insufficiency, and sufficiency. Parameters including self-reported sitting time were analyzed between each group. Pearson correlation coefficients were calculated to explore the relationships of serum 25-hydroxy vitamin D levels with age-adjusted risk factors. A scatter plot demonstrated the relationship between serum 25-hydroxy vitamin D levels and self-reported sitting time. The association between serum 25-hydroxy vitamin D levels and self-reported sitting time was assessed by multivariate linear regression with adjustment for age, sex, waist circumference, low-density lipoprotein, triglycerides, and smoking and drinking status. (3) Results: We analyzed the data from 396 participants. A total of 41.4% of participants were male, and the average age of all participants was 64.91 (±8.80) years. The participants in the high serum 25-hydroxy vitamin D group were more likely to have shorter self-reported sitting time. Serum 25-hydroxy vitamin D was negatively correlated (Pearson’s r) with self-reported sitting time, even after adjustment for age. According to the results of multivariate linear regression, vitamin D levels showed a negative association with self-reported sitting time (β = −0.131, p = 0.006) after adjustment for age, sex, waist circumference, low-density lipoprotein, triglycerides, and smoking and drinking status. (4) Conclusions: According to our research, self-reported sitting time was inversely correlated with serum 25-hydroxy vitamin D in middle-aged and elderly people in Taiwan. Meanwhile, longer self-reported sitting time can be an independent risk factor for lower serum 25-hydroxy vitamin D levels.