Aquaculture production is expected to increase with the help of genomic selection (GS). The possibility of performing GS using only a small number of SNPs has been examined in order to reduce genotyping costs; however, the practicality of this approach is still unclear. Here, we tested whether the effects of reducing the number of SNPs impaired the prediction accuracy of GS for standard length, body weight, and testes weight in the tiger pufferfish (Takifugu rubripes). High values for predictive ability (0.563–0.606) were obtained with 4000 SNPs for all traits under a genomic best linear unbiased predictor (GBLUP) model. These values were still within an acceptable range with 1200 SNPs (0.554–0.588). However, predictive abilities and prediction accuracies deteriorated using less than 1200 SNPs largely due to the reduced power in accurately estimating the genetic relationship among individuals; family structure could still be resolved with as few as 400 SNPs. This suggests that the SNPs informative for estimation of genetic relatedness among individuals differ from those for inference of family structure, and that non-random SNP selection based on the effects on family structure (e.g., site-FST, principal components, or random forest) is unlikely to increase the prediction accuracy for these traits.