This study aimed to analyze the relationship of physical activity in different domains with cardiac autonomic modulation in adults. A sample of 252 adults was randomly selected, with mean age of 42.1 (± 16.5) years, being 58% of women. Cardiac autonomic modulation was assessed through indexes of heart rate variability in time (SDNN, RMSSD) and frequency (LF, HF) domains for linear method, and by Poincaré plot for non-linear method (SD1, SD2 components). Domains of PA (occupation, sport, leisure time/commuting, and total) were assessed by Baecke’s questionnaire. Variables of age, gender, socioeconomic status (questionnaire) and body mass index (objectively measures) were covariates. Occupational PA was positively related to LF (β = 2.39, 95% CI 0.24; 4.54), sports PA was positively related to SDNN (β = 3.26, 95% CI 0.18; 7.05), RMSSD (β = 4.07, 95% CI 0.31; 7.85), and SD1 (β = 2.85, 95% CI 0.11; 5.81), and leisure time/commuting PA was positively related to SDNN (β = 3.36, 95% CI 0.28; 6.70) and RMSSD (β = 3.53, 95% CI 0.46; 7.52) indexes. Total PA was related to RMSSD (β = 1.70, 95% CI 0.04; 3.72). Sports, leisure time/commuting, and total PA were related to higher parasympathetic modulation, while occupational PA was related to higher sympathetic modulation to the heart in adults.