Camellia oleifera, an important tree species and source of edible oil in China, has received significant attention owing to the oil’s high unsaturated fatty acid content, which has benefits for human health. However, the mechanisms underlying C. oleifera yield and oil quality are largely unknown. In this study, 180 F1 progenies were obtained from two parents with obvious differences in fruit- and oil-related traits. We constructed a high-density genetic map using a double digest restriction site-associated DNA sequencing (ddRAD-Seq) strategy in C. oleifera. This map spanned 3327 cM and anchored 2780 markers in 15 linkage groups (LGs), with an average marker interval of 1.20 cM. A total of 221 quantitative trait loci (QTLs) associated with fruit- and oil-related traits were identified across three years’ worth of phenotypic data. Nine QTLs were detected simultaneously in at least two different years, located on LG02, LG04, LG05, LG06, and LG11, and explained 8.5–16.6% of the phenotypic variation in the corresponding traits, respectively. Seventeen major QTLs were obtained that explained 13.0–16.6% of the phenotypic variance. Eleven and five flanking SNPs of major QTLs for fruit- and oil-related traits were detected which could be used for marker-assisted selection in C. oleifera breeding programs. Furthermore, 202 potential candidate genes in QTL regions were identified based on the collinearity of the genetic map and the C. oleifera “CON” genome. A potential regulatory network controlling fruit development and oil biosynthesis was constructed to dissect the complex mechanism of oil accumulation. The dissection of these QTLs will facilitate the gene cloning underlying lipid synthesis and increase our understanding in order to enhance C. oleifera oil yield and quality.