The impact of imperfections, which are always present on surfaces of lenses with centimeter-size curvature radii, on the Casimir force in the lens-plate geometry is investigated. It is shown that the commonly used formulation of the proximity force approximation is inapplicable for spherical lenses with surface imperfections, such as bubbles and pits. More general expressions for the Casimir force are derived that take surface imperfections into account. Using these expressions we show that surface imperfections can both increase and decrease the magnitude of the Casimir force up to a few tens of percent when compared with the case of a perfectly spherical lens. We demonstrate that the Casimir force between a perfectly spherical lens and a plate described by the Drude model can be made approximately equal to the force between a sphere with some surface imperfection and a plate described by the plasma model, and vice versa. In the case of a metallic sphere and semiconductor plate, approximately the same Casimir forces are obtained for four different descriptions of charge carriers in the semiconductor if appropriate surface imperfections on the lens surface are present. The conclusion is made that there is a fundamental problem in the interpretation of measurement data for the Casimir force, obtained by using spherical lenses of centimeter-size radii, and their comparison with theory.