Delays in diagnosing breast cancer (BC) can lead to poorer outcomes. We investigated factors related to the diagnostic interval in a population-based cohort of 3202 women diagnosed with BC in Queensland, Australia. Interviews ascertained method of detection and dates of medical/procedural appointments, and clinical information was obtained from medical records. Time intervals were calculated from self-recognition of symptoms (symptom-detected) or mammogram (screen-detected) to diagnosis (diagnostic interval (DI)). The cohort included 1560 women with symptom-detected and 1642 with screen-detected BC. Symptom-detected women had higher odds of DI of >60 days if they were Indigenous (OR = 3.12, 95% CI = 1.40, 6.98); lived in outer regional (OR = 1.50, 95% CI = 1.09, 2.06) or remote locations (OR = 2.46, 95% CI = 1.39, 4.38); or presented with a "non-lump" symptom (OR = 1.84, 95% CI = 1.43, 2.36). For screen-detected BC, women who were Indigenous (OR = 2.36, 95% CI = 1.03, 5.80); lived in remote locations (OR = 2.35, 95% CI = 1.24, 4.44); or disadvantaged areas (OR = 1.69, 95% CI = 1.17, 2.43) and attended a public screening facility (OR = 2.10, 95% CI = 1.40, 3.17) had higher odds of DI > 30 days. Our study indicates a disadvantage in terms of DI for rural, disadvantaged and Indigenous women. Difficulties in accessing primary care and diagnostic services are evident. There is a need to identify and implement an efficient and effective model of care to minimize avoidable longer diagnostic intervals.