Excessive sedentary time (SED) and long SED bouts are associated with cardiovascular diseases (CVD) and increased mortality. Low heart rate variability (HRV), indicating autonomic dysfunction, increases mortality and CVD morbidity. Information about the association between prolonged SED and HRV is lacking. The aim was to assess the relationship between SED bouts and HRV. Physical activity (PA), SED, HRV, and cardiorespiratory fitness (CRF) were collected from a birth‐cohort sample (n = 4150) at 46 years. PA and SED were measured for 14 days with an activity monitor (Polar Active, Polar Electro, Finland). SED accumulating in bouts of at least 30 and 60 minutes (SED30, SED60) was calculated. Linear regression was used to study the relationship between prolonged SED and HRV accounting for CRF, PA, and health covariates. Higher SED60 and in women SED30 were associated with higher root mean square of differences in R‐R intervals (rMSSD) after adjustments (β = .082‐.104). In women, higher SED60 was associated with lower ratio between low‐ and high‐frequency powers (β = −.060). Sedentary bouts were not associated with resting HR or post‐exercise HR recovery. A positive relationship between SED bouts and rMSSD independent of PA and CRF was found, prolonged SED being positively associated with cardiac parasympathetic activity in midlife.