The Young’s modulus and Poisson’s ratio determined for the rocks under uniaxial or triaxial loading conditions represent necessary input parameters for solving many geotechnical tasks. However, these effective stiffness moduli, particularly the Poisson’s ratios, measured on different samples sometimes substantially differ, even for visually compact and macroscopically homogeneous rock. This has been observed for Brenna sandstone, especially under conditions of conventional triaxial extensions. The aim of this study was, thus, to reveal differences in rock structure that could cause such behavior. Several complementary methods were used to investigate the structure of this sandstone: X-ray computed tomography (CT), visual analysis using optical scanning and stereo-microscopy, and a new method combining water jet erosion with these visualization techniques. Analyses of this structure revealed mechanically more resistant compact thick and adjacent thin weak layers. In rock samples, these layers have different patterns and orientations. To demonstrate the influence of layer orientation on effective stiffness moduli measured under different loading conditions, FEM calculations were performed for idealized structural models. The outcomes of this numerical analysis are in qualitative agreement with the results of the loading experiments and the layered structure revealed in the samples. The ability of the proposed method using water erosion to analyze the rock structure in detail was verified. The method offers an interesting alternative to standard visual and X-ray CT techniques. The numerical calculations indicate the importance of latent layered defects in sedimentary rocks for fine measurements of stiffness moduli (especially the Poisson’s ratios) used as inputs for geotechnical applications.