In hyperarid deserts, endolithic microbial communities colonize the rocks' interior as a survival strategy. Yet, the composition of these communities and the drivers promoting their assembly are still poorly understood. We analysed the diversity and community composition of endoliths from four different lithic substrates - calcite, gypsum, ignimbrite and granite - collected in the hyperarid zone of the Atacama Desert, Chile. By combining microscopy, mineralogy, spectroscopy and high throughput sequencing, we found these communities to be highly specific to their lithic substrate, although they were all dominated by the same four main phyla, Cyanobacteria, Actinobacteria, Chloroflexi and Proteobacteria. Our finding indicates a fine scale diversification of the microbial reservoir driven by substrate properties. The data suggest that the overall rock chemistry and the light transmission properties of the substrates are not essential drivers of community structure and composition. Instead, we propose that the architecture of the rock, i.e., the space available for colonization and its physical structure, linked to water retention capabilities, is ultimately the driver of community diversity and composition at the dry limit of life.