Background: Limited reliability data exist for evaluation of spinal edema changes on magnetic resonance imaging (MRI) with short tau inversion recovery (STIR) sequences. Purpose: To assess the inter-observer reliability for evaluation of STIR signal increase related to Modic changes (MCs) on MRI of the lumbar spine. Material and Methods: We prospectively included 120 patients imaged to confirm their eligibility for the AIM (Antibiotics In Modic changes) trial. Three experienced radiologists independently evaluated MCs on T1-/T2-weighted fast spin-echo images and subsequently MC-related STIR signal increases. Inter-observer reliability was analyzed at four endplates (L4-S1) by calculating kappa values and means of differences with 95% limits of agreement. Results: Overall agreement (mean Fleiss' kappa for all endplates and observers) was very good for presence of STIR signal increase (0.86), and moderate for its categorized height (0.51), anteroposterior extent (0.48), and volume (0.56). For height of region with STIR signal increase measured in % points of vertebral body height, the largest mean of differences was 6.9 and widest range for limits of agreement was AE22.3 for all endplates combined. The corresponding numbers were 11.2 AE 34.5 for anteroposterior extent of the STIR signal increase measured in % points of anteroposterior endplate diameter and 0.9 AE 7.6 for its maximum measured intensity on a % point scale (0% ¼ normal vertebral marrow intensity, 100% ¼ cerebrospinal fluid intensity). Conclusion: Inter-observer reliability was very good for the presence and intensity of MC-related STIR signal increases, and moderate for their size.