Background: Modic changes (MC) are associated with low back pain (LBP), but effective treatments are lacking. The aim of this randomized, placebo-controlled, double-blinded trial was to evaluate the efficacy of zoledronic acid (ZA) for chronic LBP among patients with MC in magnetic resonance imaging (MRI). Methods: Inclusion criteria were LBP lasting ≥3 months, with an intensity of ≥6 on a 10-cm VAS or an Oswestry Disability Index (ODI) of ≥30%, and MC in MRI. Patients were randomized into single intravenous infusion of ZA 5 mg (n = 20), or placebo (n = 20) groups. The primary outcome was LBP intensity, secondary outcomes leg pain intensity, ODI, health-related quality of life (RAND-36), lumbar flexibility, sick leaves and use of pain medication. The treatment differences at one month and one year were analysed using ANCOVA with adjustment for the baseline score. Results: The mean difference (MD) between the groups in the primary outcome, intensity of LBP, was 1.4 (95% confidence intervals (CI) 0.01 to 2.9) in favour of ZA at one month. We observed no significant between-group difference in the intensity of LBP at one year (MD 0.7; 95% CI −1.0 to 2.4) or in secondary outcomes at any time point except that 20% of patients in the ZA group used non-steroidal anti-inflammatory drugs at one year compared to 60% in the placebo group (P = 0.022). Acute phase reactions (fever, flu-like symptoms, arthralgia) emerged in 95% of the patients in the ZA group, compared to 35% in the placebo group. Conclusions: ZA was effective in reducing the intensity of LBP in the short term and in reducing the use of NSAIDs within the time span of one year among patients with chronic LBP and MC confirmed in MRI. Although the results seem encouraging, larger studies are required to analyse the effectiveness and safety of ZA for patients with MC. Trial registration: ClinicalTrial.gov identifier NCT01330238.