The fragmentomics-based cell-free DNA (cfDNA) assays have recently demonstrated remarkable capabilities in identifying various cancers from non-conditional healthy controls. However, their accuracy in differentiating early-stage cancers from benign lesions, especially with inconclusive imaging results, remains unclear. In breast cancer, imaging-based screening methods often yield high false-positive rates, particularly in women with breast nodules, leading to unnecessary biopsies and increasing patient discomfort and healthcare burden. In this multi-center study, we enrolled 560 female participants and showed that cfDNA fragmentomics, using whole-genome sequencing, is a strong non-invasive biomarker for breast cancer. Among various cfDNA fragmentomics profiles, the fragment size ratio (FSR), fragment size distribution (FSD), and copy number variation (CNV) exhibited superior distinguishing ability compared to Griffin, motif breakpoint (MBP), and neomer features. Our cfDNA fragmentomics (cfFrag) model, which incorporates the three optimal fragmentomics features, successfully discriminated early-stage breast cancers from benign nodules, even at a low sequencing depth (3x). Notably, it demonstrated a specificity of 94.1% in asymptomatic healthy women at a 90% sensitivity for breast cancer. Furthermore, we highlight the clinical utility of the cfFrag model in predicting patient responses to neoadjuvant chemotherapy (NAC), and its integration with multimodal features, including radiological results and cfDNA methylation features, achieved high AUC values of 0.93-0.94 and 0.96, respectively.