The flowers, leaves, seed cakes and fruit shells of Camellia oleifera are rich in bioactive polysaccharides, which can be used as additives in food and other industries. In this study, a Box−Behnken design was used to optimize the extraction conditions of polysaccharides from C. oleifera flowers (P-CF), leaves (P-CL), seed cakes (P-CC), and fruit shells (P-CS). Under the optimized extraction conditions, the polysaccharide yields of the four polysaccharides were 9.32% ± 0.11 (P-CF), 7.57% ± 0.11 (P-CL), 8.69% ± 0.16 (P-CC), and 7.25% ± 0.07 (P-CS), respectively. Polysaccharides were mainly composed of mannose, rhamnose, galacturonic acid, glucose, galactose, and xylose, of which the molecular weights ranged from 3.31 kDa to 128.06 kDa. P-CC had a triple helix structure. The antioxidant activities of the four polysaccharides were determined by Fe2+ chelating and free radical scavenging abilities. The results showed that all polysaccharides had antioxidant effects. Among them, P-CF had the strongest antioxidant activity, of which the highest scavenging ability of DPPH•, ABTS•+, and hydroxyl radical could reach 84.19% ± 2.65, 94.8% ± 0.22, and 79.97% ± 3.04, respectively, and the best chelating ability of Fe2+ could reach 44.67% ± 1.04. Overall, polysaccharides extracted from different parts of C. oleifera showed a certain antioxidant effect, and could be developed as a new type of pure natural antioxidant for food.