Detecting community-wide statistical relationships from targeted amplicon-based and metagenomic profiling of microbes in their natural environment is an important step toward understanding the organization and function of these communities. We present a robust and computationally tractable latent graphical model inference scheme that allows simultaneous identification of parsimonious statistical relationships among microbial species and unobserved factors that influence the prevalence and variability of the abundance measurements. Our method comes with theoretical performance guarantees and is available within the SParse InversE Covariance estimation for Ecological ASsociation Inference (SPIEC-EASI) framework ('SpiecEasi' R-package). Using simulations, as well as a comprehensive collection of amplicon-based gut microbiome datasets, we illustrate the method's ability to jointly identify compositional biases, latent factors that correlate with observed technical covariates, and robust statistical microbial associations that replicate across different gut microbial data sets.