Solid contacts involving soft materials are important in mechanical engineering or biomechanics. Experimentally, such contacts have been shown to shrink significantly under shear, an effect which is usually explained using adhesion models. Here we show that quantitative agreement with recent high-load experiments can be obtained, with no adjustable parameter, using a non-adhesive model, provided that finite deformations are taken into account. Analysis of the model uncovers the basic mechanisms underlying shear-induced area reduction, local contact lifting being the dominant one. We confirm experimentally the relevance of all those mechanisms, by tracking the shear-induced evolution of tracers inserted close to the surface of a smooth elastomer sphere in contact with a smooth glass plate. Our results suggest that finite deformations are an alternative to adhesion, when interpreting a variety of sheared contact experiments involving soft materials.