Influence line (IL) has emerged as very promising damage indices for bridge damage detection. This study proposed a method to localize and quantify damage in beam structures by estimating section flexibility change from deflection IL (DIL) change. To this end, the relationship between second derivative of DIL change and flexibility change was established. To remove noise interference in measurement, piecewise quadratic functions were used to fit and replace noisy DIL change curves, wherein the coefficients of quadratic function were determined via a sparse regularization method, considering the sparsity nature of damage that typically takes place in only a limited number of elements. The feasibility and accuracy of the proposed method are verified through numerical examples and laboratory experiments. Through four hypothetical damage scenarios of a simply supported beam with one or two damaged locations, its ability to quantify minor damage and its anti-noise robustness were well verified. Finally, a laboratory experiment on a simply supported aluminum beam illustrated that the location and extent of damage could be successfully identified in the single-damage and double-damage cases. The numerical and experimental results indicate that the proposed method is promising for future damage localization and quantification of bridge structures.