The mechanical behavior of mudstones is significantly affected by their mesostructure. This study investigates the damage evolution of red mudstone mesostructures under uniaxial compression through U-Net image segmentation, mesoscopic representative elementary area (mREA), meso-element equivalent method, and linearly superimposed model. Results show that the mREAs can be simplified as a binary structure comprising nonclayey minerals and a porous matrix. The inclusion-matrix interfaces constituted by the dominant inclusions, such as large inclusions and a series of inclusions with specific arrangements, are the weakest regions at the postpeak stage. The shear failure of these interfaces leads to general shear failure with a penetrating principal fracture or local shear with several nonpenetrating local fractures in the mREAs. The damage evolution and failure mode are sensitive to the inclusion geometric features, e.g., inclination, grain size, and spacing. However, the uniaxial compressive strength (UCS) is mainly determined by the included angle between the dominant inclusion inclination and the loading direction. With consistent composition, an increase in the included angle from 0 to 90° results in an increase in the UCS of the muddy facies and a decrease in the sandy facies, with the UCS of the sandy facies being more sensitive to changes in the included angle. Our findings indicate that the structural failure of red mudstones from mesoscopic to macroscopic is driven by the accumulation and expansion of shear failure at multiple scales due to the presence of hard–soft interfaces. This study enhances our understanding of the structural failure mechanism of mudstones and serves as a practical reference for the mesomechanical testing of clay rocks.