Frictional failure is the dominant deformation mechanism for rocks in the upper crust while in the middle crust rocks begin to deform viscously. Within this transition, brittle and viscous phases coexist, forming semi‐frictional materials. While semi‐frictional deformation on large scales might play an important role in understanding the transition between earthquakes and slow slip/creep, it can also be observed at smaller scales. Here, we use field observations of the Papoose Flat pluton in eastern California to study deformation of heterogeneous materials during shearing. Clast concentration varies between 2% and 12% by area. Field and microscopic observations show that the matrix deforms viscously, while the clasts fail in a brittle manner. We systematically document clast concentration and spacing with respect to clast fracturing and observe increasing frictional failure of clasts with increasing clast concentration. To test which matrix viscosities impose enough stresses on the clasts to lead to frictional deformation, we complement field observations with 2D numerical models. Maps with 7% by area randomly placed circular clasts are created and deformed under simple shear kinematic conditions. We test different matrix viscosities, from constant low and high viscosity (1017 and 1019 Pa.s, respectively), to dislocation creep for granite. Clasts in the vicinity of other clasts are affected by stresses around their neighbors. This effect decreases with increasing clast distance. Our field observations and numerical results suggest that the viscous phase can impose significant stresses onto the brittle phase, causing failure even at very low clast concentrations and in the absence of clast‐clast interactions.