As porous, heterogeneous, and anisotropic material, the microscopic structure of the rock has a significant influence on its mechanical properties. Rare studies were devoted to this area using pore scale modeling and simulations. In this paper, different types of sandstones are imaged using micro-CT technology. The rock porosity is obtained by filtering, binarization, and threshold segmentation. The texture coefficient (TC) and the tortuosity of the rock skeleton are calculated by open source program, where the tortuosity of the rock skeleton is firstly used to characterize the microscopic structure of the rock. Combining with the rock mechanics parameters obtained in the laboratory, the simulation of uniaxial compression is performed on the reconstructed pore scale rock finite element mesh model by ANSYS software. Young’s modulus, compressive strength, yield strength, shear modulus, and other related parameters obtained by numerical simulation are adopted to determine the optimal representative volume element (RVE) size. Moreover, the effects of microscopic structure characteristics on the mechanical properties of the rock are studied quantitatively. The results indicate that the averaged von Mises stress distribution, displacement field, and plastic strain field of rocks show anisotropy and heterogeneity. The stress concentration and the X-shaped conjugate plastic shear zone are investigated. The samples of S1∼S4 reach the elastic limit and enters the plastic yield state, when the strain is about 0.5%. And the critical yield strain of samples S5300-1∼S5400-2 is about 1%. Then, the quantitative relationships between porosity, TC, tortuosity of rock skeleton and rock mechanics parameters of digital rock samples are established and analyzed. The tortuosity of the rock skeleton is highly correlated with the mechanical parameters of the rock, i.e., Young’s modulus (R2 = 0.95), compressive strength (R2 = 0.94), yield strength (R2 = 0.92), and shear modulus (R2 = 0.94), which is believed to be more feasible to reveal the impacts of the microstructure of the rock on its mechanical properties.