High-quality sleep is an important factor in sustaining health and improving well-being. Previous evidence has demonstrated the positive associations between increased physical activity and reduced sedentary behaviour (SB) with sleep quality. The substitutional relationships between SB, light-intensity physical activity (LPA), and moderate-to-vigorous physical activity (MVPA) need to be considered when examining how a particular behaviour may impact sleep quality. No studies, to our knowledge, have explored these substitutional relationships in middle-aged adulthood. Using an isotemporal substitution approach, this study examined the associations of replacing sedentary time with physical activity on sleep quality measures in a sample of middle-aged adults in Japan. Data from 683 adults aged 40–64 living in Japan were used. The average daily time spent in SB, LPA, and MVPA was objectively assessed by accelerometers. Two self-reported sleep quality measures were obtained using questionnaires, including rest by sleep and sleep quality. Multivariable linear regression models were used to assess the associations of SB, LPA, and MVPA with the sleep quality measures stratified by gender. We found that each 60 min unit of SB or LPA replaced with MVPA was favourably associated with rest by sleep among women (β = 0.16, 95% CI 0.07, 0.28, p < 0.001; β = 0.18, 95% CI 0.07, 0.32, p < 0.05, respectively). There were no significant associations between SB, LPA, and MVPA with sleep measures in men across all three models. These findings indicate that higher MVPA has a positive association with sleep quality in middle-aged women.