The choice of which covariates to include in a Genome-Wide Association Study (GWAS) is important since it affects the ability to detect true association signal of variants, to correct for confounders and avoid false positives, and the running time of the analysis. Commonly used covariates include age, sex, genotyping batches, genotyping array type, as well as an arbitrary number of Principal Components (PCs) used to adjust for population structure. Despite the importance of this issue, there is no consensus or clear guidelines for the right choice of covariates. Therefore, studies typically employ heuristics for their choice with no clear justification. Here, we explore the dependence of the GWAS analysis results on the choice of covariates for a wide range of quantitative and binary human phenotypes. We propose guidelines for covariates choice based on the phenotype's type (quantitative vs. disease), the heritability, and the disease prevalence, with the goal of maximizing the statistical power to detect true associations and fit accurate polygenic scores while avoiding spurious associations and minimizing computation time. We analyze 36 traits in the UK-Biobank dataset. We show that the genotype batch and assessment center can be safely removed as covariates, thus significantly reducing the GWAS computational burden for these traits.